WO2015139119A1

WO2015139119A1 - System and method for validating data source input from a crowd sourcing platform

Info

Publication number: WO2015139119A1
Application number: PCT/CA2015/000169
Authority: WO
Inventors: Mark Gordon MCALLISTER; Michael Patrick KERRY; Martin David Brian KAYE
Original assignee: Verosource Solutions Inc.
Priority date: 2014-03-19
Filing date: 2015-03-19
Publication date: 2015-09-24

Abstract

The present disclosure relates to systems and methods for collecting, validating and displaying data from data sources as validated by crowd sourcing and data metrics. The method includes steps of receiving a plurality of data items and a plurality of attributes for each data item, each attribute having one or more instances, each instance having an Attribute Value; receiving from a display interface, a query to display a portion of the plurality of attributes; receiving from the display interface, one or more votes from one or more users for one or more Attribute Values for one or more of the plurality of data items; determining a rank or score of reliability for each of the one or more Attribute Values based on the one or more votes received and data metrics; and providing a highest ranked or scored Attribute Value for each attribute to the display interface for display.

Description

SYSTEM AND METHOD FOR VALIDATING DATA SOURCE INPUT FROM A

CROWD SOURCING PLATFORM

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This non-provisional patent application claims the benefit of and priority from U.S. provisional patent application No. 61/955,448 filed March 19, 2014, which is incorporated herein by reference in its entirety.

FIELD

[0002] Embodiments described herein relate generally to systems and methods for validating data from a crowd sourcing platform, and more particularly to systems and methods for collecting and providing information and services to a network based on information validated in real-time by data metrics and the crowd sourcing platform.

BACKGROUND

[0003] Organizations from many different industries generate valuable information for business purposes. For example, travel agents or individuals use information about hotels and also about businesses that cater to tourists to help travellers plan vacations, banks and other financial institutions use information about local businesses to inform their marketing activities and mobile search applications need the latest business contact information to direct users to the services they desire. In general, organizations use the information gathered to provide services or to market their services to new customers, and it is an integral part of the success of their business.

[0004] While many organizations generate business data from static sources, such as the examples above, in today's fast-paced market it is becoming increasingly advantageous to utilize data generated in real-time. In addition, for certain services, generating useful data from static sources is infeasible. This has led many organizations to leverage dynamic data sources, such as their customer base, for information. Examples include news media providers, traffic or weather services and other purveyors of real-time data.

[0005] The nature of dynamic information often makes collection from a large distributed network of users or customers ideal. As a result applications have arisen for crowd sourcing information or user generated content. Examples such as the online encyclopedia Wikipedia, or the social news sharing website Reddit show that information can be contributed by users in nearly real-time. A recent trend has emerged where organizations rely nearly exclusively on their users for data and content. Online and mobile organizations in particular are relying on user-generated data to provide services. Examples include ride-sharing company Uber, the distributed errand service TaskRabbit and traffic adviser Waze. In addition, crowd-sourcing platforms are being used more and more to disseminate critical information in times of crisis, such as the use of the FireEye application during the 2014 Hong Kong protests. In such situations the accuracy of collected data is paramount.

[0006] In general, organizations which rely on dynamic data sources to provide services may be particularly dependent on the reliability of collected data. The usefulness of collected information is contingent on its accuracy. An organization that collects and uses inaccurate data may provide less valuable services or be less attractive to potential customers. The accuracy of collected data may be dependent on many factors, such as location, or the value of the data itself. In addition, the accuracy or truthfulness of information may fluctuate with time, making information gathered one second potentially invalid the next. For example Google™ and other search engines have catalogued much of the world's locations and related information. Despite this, information on many locations remains at times inaccurate or incomplete. Information collected via crowd sourcing may be particularly prone to inaccuracy. While users may provide a wealth of potentially relevant data, in practice different user perspectives, subjective views and issues with human-computer interfaces often lead to unreliable data. Without proper validation, the relevancy of dynamic data cannot be properly established, rendering the information significantly less valuable.

[0007] Unfortunately data collected from dynamic data sources is often difficult to validate. The dynamic nature of the data, lack of control over the data source and the inconsistency and unreliability of available metadata make validating incoming information challenging. Metadata inconsistency between different sources also makes it hard to aggregate and collate data from different sources. Different dynamic data sources may also provide data of significantly different average accuracy, making aggregated data sets yet more difficult to validate. For example, medical facility published wait times fluctuate frequently and vary widely depending on a number of different factors, making it difficult to validate without being at the location. The result is that in many places this data is either extremely unreliable or unavailable altogether.

[0008] With the ubiquity of dynamic data sources such as crowd sourcing platforms, applications and social media, assessing the validity of real-time information is a problem. Due to validation difficulties, information collected in this way is often relegated to static data, such as that related to a particular location, or items where only the aggregate of the data is important, such as popularity contests. Organizations that attempt to validate dynamic data have done so on an ad-hoc basis and with limited success. Information which cannot be validated may be unused altogether. The example of medical facility data is again apt: business listing services such as Google Places™, Yelp™ and the Yellow Pages™ have been unable to generate accurate wait time data, so it is simply not available. The vast majority of data available through dynamic data sources is thus left untouched since its accuracy cannot be ascertained.

[0009] There exists a need to inform collection and dynamically validate fluctuating real-time data sources, including data sources of questionable reliability. In view of these example shortfalls of known approaches, there exists a need for systems, platforms, devices and methods for validating or verifying data, or at least alternatives.

SUMMARY

[00010] The present disclosure relates to a system and method for collecting and assessing the reliability of real-time data using data metrics and crowd-sourced consensus.

[00011] In one aspect, a computer-network system for collecting, validating and displaying information of a plurality of data items is disclosed, the system includes: a data storage device for collecting and storing, from a plurality of data sources in real-time, a plurality of data items and a plurality of attributes for each of the plurality of data items, each attribute having one or more instances, each instance having an Attribute Value; a display interface to display on a display device a portion of the plurality of attributes based on a query; and a server implemented by at least one processor, the server configured to: a) receive, from the display interface, one or more votes from one or more users for one or more attributes of the portion of the plurality of attributes; b) determine a rank or score of reliability for each of the one or more attributes based on the one or more votes received, the one or more instances, and the Attribute Values; and c) provide a highest ranked or scored attribute to the display interface for display.

[00012] In another aspect of the system, each Attribute Value has a corresponding data source of the plurality of data sources; and the server is configured to determine a rank or score of reliability for each of the plurality of data sources, and determine the rank or score of reliability for each of the one or more attributes based on the rank or score of reliability for the corresponding data source.

[00013] In another aspect of the system, the server configured to receive, from the display interface, one or more votes from one or more users for one or more Attribute Values, determine a rank or score of reliability for each of the one or more Attribute Values based on the one or more votes received, and provide a highest ranked or scored Attribute Value of each attribute to the display interface for display.

[00014] In one aspect, computer-network-implemented method for collecting, validating and displaying information of a plurality of data items is disclosed, the method includes: a) receiving from a plurality of data sources in real-time, and storing in a data storage device, a plurality of data items and a plurality of attributes for each of the plurality of data items, each attribute having one or more instances, each instance having an Attribute Value; b) receiving from a display interface, a query to display a portion of the plurality of attributes; c) receiving, from the display interface, one or more votes from one or more users for one or more Attribute Values for one or more of the plurality of data items; d) determining, by a server implemented by a processor, a rank or score of reliability for each of the one or more Attribute Values based on the one or more votes received; and e) providing, by the server, a highest ranked or scored Attribute Value for each attribute in the portion of the plurality of attributes to the display interface for display. In another aspect of the method, each Attribute Value has a corresponding data source of the plurality of data sources, and the method further includes determining a rank or score of reliability for each of the plurality of data sources, and determining the rank or score of reliability for each of the one or more Attribute Values based on the rank or score of reliability for the corresponding data source.

[00015] In yet another aspect of the system, the server is configured to determine the rank or score of reliability for an Attribute Value of the one or more Attribute Values based on a rank or score of reliability associated with another Attribute Value of an instance of the same attribute.

[00016] In yet another aspect of the method, the method can determine the rank or score of reliability for an Attribute Value of the one or more Attribute Values based on a rank or score of reliability associated with another Attribute Value of an instance of the same attribute.

[00017] In a further aspect of the system, the server configured to determine the rank or score of reliability for each of the one or more attributes based on the rank or score of reliability for each of the one or more Attribute Values of each instance of the respective attribute.

[00018] In yet another aspect of the system, each attribute of the one or more attributes is associated with a data item and a category, and the server is configured to determine the rank or score of reliability for each of the one or more attributes in relation to other attributes associated with the same data item and the same category.

[00019] In yet another aspect of the method, each attribute of the one or more Attribute Values is associated with a data item and a category, and the method further include determining the rank or score of reliability for each of the one or more Attribute Values in relation to other Attribute Values associated with the same data item and the same category.

[00020] In one aspect of the system, the server is configured to determine a rank or score of reliability for each of the one or more users based on indicia of quality of contributed content or votes of the respective user, and determine the rank or score of reliability for each of the one or more attributes by weighting any vote received from the respective user based on the rank or score of reliability for the respective user.

[00021] In another aspect of the method, the method can determine a rank or score of reliability for each of the one or more users based on indicia of quality of contributed content or votes of the respective user, and determining the rank or score of reliability for each of the one or more Attribute Values by weighting any vote received from the respective user based on the rank or score of reliability for the respective user.

[00022] In another aspect of the system, the server is configured to implement machine learning techniques to determine, for at least one of the one or more attributes, a system confidence value independent of voting, and determine the rank or score of reliability for the at least one attribute based on the system confidence value. [00023] In yet another aspect of the system, the server is configured to implement machine learning techniques to determine, for at least one of the one or more Attribute Values, a system confidence value independent of voting, and determine the rank or score of reliability for the at least one Attribute Value based on the system confidence value.

[00024] In one aspect of the method, the method can implement machine learning techniques to determine, for at least one of the one or more Attribute Values, a system confidence value independent of voting, and determine the rank or score of reliability for the at least one Attribute Value based on the system confidence value.

[00025] In still another aspect of the system, the server is configured to determine the rank or score of reliability for each of the one or more attributes by comparing the one or more Attribute Values for each instance of the respective attribute.

[00026] In a further aspect of the system, each of the one or more attributes is associated with a category, and the server is configured to determine a rank or score of reliability for each of the one or more attributes based on a rank or score of reliability associated with another attribute of a same or similar category.

[00027] In one aspect of the system, an attribute of the one or more attributes is associated with an elapsed time since its collection, and the server is configured to determine a rank or score of reliability for the attribute based on the elapsed time since its collection.

[00028] In another aspect of the system, an Attribute Value of an instance of an attribute of the one or more attributes is associated with an elapsed time since its collection, and the server is configured to determine a rank or score of reliability for the Attribute Value based on the elapsed time since its collection.

[00029] In another aspect of the method, an Attribute Value of an instance of an attribute of the one or more attributes is associated with an elapsed time since its collection, and the method further include determining a rank or score of reliability for the Attribute Value based on the elapsed time since its collection.

[00030] In yet another aspect of the system, each vote is associated with a decay value, and the server is configured to determine the rank or score of reliability for each of the one or more attributes by weighting the respective vote with the associated decay value.

[00031 ] In another aspect of the method, each vote may be associated with a decay value, and the method further include determining the rank or score of reliability for each of the one or more Attribute Values by weighting the respective vote with the associated decay value.

[00032] In still another aspect of the system, each of the one or more attributes is associated with metadata comprising a data source, a location of collection and a time of collection, and the server is configured to determine the rank or score of reliability for each of the one or more attributes based on one or more of the data source, the location of collection and the time of collection.

[00033] In another aspect of the system, the location of collection has a positive impact on the rank of score of reliability for each of the one or more attributes if the location of collection is within a predetermined proximity to a location of the data item associated with the respective attribute.

[00034] In yet another aspect of the system, each of the one or more Attribute Values is associated with metadata comprising a data source, a location of collection and a time of collection, and the server is configured to determine the rank or score of reliability for each of the one or more Attribute Values based on one or more of the data source, the location of collection and the time of collection.

[00035] In yet another aspect of the method, each of the one or more Attribute Values is associated with metadata comprising a data source, a location of collection and a time of collection, and the method further include determining the rank or score of reliability for each of the one or more Attribute Values based on one or more of the data source, the location of collection and the time of collection.

[00036] In a further aspect of the system, the location of collection has a positive impact on the rank or score of reliability for each of the one or more Attribute Values if the location of collection is within a predetermined proximity to a location of the data item associated with the respective attribute for the respective Attribute Value.

[00037] In still another aspect of the method, the location of collection has a positive impact on the rank or score of reliability for each of the one or more Attribute Values if the location of collection is within a predetermined proximity to a location of the data item associated with the respective attribute for the respective Attribute Value.

[00038] In one aspect of the system, the server is configured to determine a rank or score of reliability for each of the plurality of data items based on the rank or score of reliability for each of the plurality of attributes for the respective data item.

[00039] In one aspect of the method, the method can determine a rank or score of reliability for each of the plurality of data items based on the rank or score of reliability for each of the one or more Attribute Values for each attribute of the respective data item.

[00040] In another aspect of the system, each vote comprises a Vote Value, the server is configured to determine or adjust the rank or score of reliability for each of the attributes based on the Vote Value.

[00041 ] In another aspect of the method, each vote includes a Vote Value, and the method further include determining or adjusting the rank or score of reliability for each of the one or more Attribute Values based on the Vote Value. [00042] In another aspect of the system and the method, at least a portion of the one or more

Attribute Values are dynamic fluctuating Attribute Values.

[00043] In still another aspect of the system, the server is configured to implement machine learning techniques to determine, for at least one attribute of the one or more attributes, one or more predicted values for the at least one attribute and determine the rank or score of reliability for the at least one attribute by comparing its Attribute Value to the one or more predicted values.

[00044] In one aspect of the system, the server is configured to implement machine learning techniques to determine, for at least one attribute, one or more predicted Attribute Values and determine the rank or score of reliability for the at least one attribute by comparing the one or more Attribute Values of each of its instances to the one or more predicted Attribute Values.

[00045] In still another aspect of the method, the method can implement machine learning techniques to determine, for at least one attribute, one or more predicted Attribute Values, and determine the rank or score of reliability for the at least one Attribute Value by comparing the one or more Attribute Values of each of its instances to the one or more predicted Attribute Values.

[00046] In another aspect of the system, at least one Attribute Value is associated with a time expiry threshold value, and the data storage device removes or otherwise disables the Attribute Value for display or ranking after expiration of time expiry threshold value.

[00047] In a further aspect of the method, at least one Attribute Value is associated with a time expiry threshold value, and the data storage device removes or otherwise disables the Attribute Value for display after expiration of time expiry threshold value.

[00048] In a further aspect of the system, at least one Attribute Value is associated with a time expiry threshold value, the at least one Attribute Value being an instance of at least one attribute, and the server is configured to determine the rank or score of reliability for each of the at least one attributes based on the time expiry threshold value of the at least one Attribute Value.

[00049] In another aspect of the method, at least one Attribute Value is associated with a time expiry threshold value, and the method further include determining the rank or score of reliability for each of the Attribute Value based on the time expiry threshold value of the at least one Attribute Value.

[00050] In still another aspect of the system, for each of the one or more users, the server is configured to track the respective user using a unique assigned token.

[00051] In one aspect of the system, at least one vote for an attribute is associated with a time expiry threshold value, and the server is configured to determine the rank or score of reliability for the respective attribute by weighting the vote based on the time expiry threshold value.

[00052] In yet another aspect of the method, at least one vote for an Attribute Value is associated with a time expiry threshold value, and the method further include determining the rank or score of reliability for the respective Attribute Value by weighting the vote based on the time expiry threshold value.

[00053] In yet another aspect of the system, each attribute is associated with a volatility factor, and the server is configured to determine the rank or score of reliability for each of the at least one attributes based on the volatility factor.

[00054] In yet another aspect of the method, each attribute is associated with a volatility factor, and the method further includes determining the rank or score of reliability for each of the Attribute Value based on the volatility factor associated with the attribute for the Attribute Value.

[00055] In another aspect of the system, the rank or score of reliability for each of the one or more attributes comprises an Attribute Trust Score.

[00056] In still another aspect of the system, each vote is either an up vote or a down vote, each vote is associated with a weight, and the server is configured to compute the Attribute Trust Score by a difference of a sum of each up vote weighted by its associated weight and a sum of each down vote weighted by its associated weight.

[00057] In one aspect of the method, each vote is either an up vote or a down vote, wherein each vote is associated with a weight, and method further includes computing the individual Attribute Trust Score by a difference of a sum of each up vote weighted by its associated weight and a sum of each down vote weighted by its associated weight.

[00058] In one aspect of the system, the rank or score of reliability for each of the one or more

Attribute Values comprises an individual Attribute Trust Score.

[00059] In another aspect of the system, each vote is either an up vote or a down vote, each vote is associated with a weight, and the server is configured to compute the individual Attribute Trust Score by a difference of a sum of each up vote weighted by its associated weight and a sum of each down vote weighted by its associated weight.

[00060] In another aspect of the system, the data storage device is configured to remove or otherwise disable an attribute for display or ranking if its associated rank or score of reliability is below a threshold value.

[00061] In yet another aspect of the system, the data storage device is configured to remove or otherwise disable an Attribute Value for display or ranking if its associated rank or score of reliability is below a threshold value.

[00062] In another aspect of the method, Attribute Value is removed from, or otherwise disabled for display or ranking, in the data storage device if the associated rank or score of reliability of the Attribute Value is below a threshold value. [00063] In still a further aspect of the system, each user or data source is associated with a trustworthiness factor based on a rank or score of reliability for the user or the data source, gamification points for the user or the data source, and historical data for the user or the data source. Attributes and attribute values (instances of attributes) can be removed from ranking or displaying aspect of embodiments described herein, but not from the data store. The system may still use past expired values to inform historical confidence values, and so on.

[00064] In yet another aspect of the method, each user or data source is associated with a trustworthiness factor based on a rank or score of reliability for the user or the data source, gamification points for the user or the data source, and historical data for the user or the data source.

[00065] In one aspect, at least two of the one or more instances are grouped into an instance group, and determining the rank or score of reliability for each of the one or more Attribute Values includes determining a rank or score of reliability for the instance group.

[00066] In another aspect, the at least two of the one or more instances are grouped based on one or more of: respective Attribute Value, respective number of votes received, respective attribute category and respective rank or score of reliability.

[00067] In one aspect, a computer-network-implemented method for collecting, validating and displaying information of a plurality of data items is disclosed. The method includes: a) receiving from a plurality of data sources in real-time, and storing in a data storage device, a plurality of data items and a plurality of attributes for each of the plurality of data items, each attribute having one or more instances, each instance having an Attribute Value; b) receiving from a display interface, a query to display a portion of the plurality of attributes; c) receiving, from the display interface, one or more votes from one or more users for one or more Attribute Values for one or more of the plurality of data items; d) grouping, by a server implemented by a processor, at least two of the one or more instances into an instance group; e) determining, by the server, a rank or score of reliability for each of the one or more Attribute Values and the instance group based on the one or more votes received; and f) providing, by the server, a highest ranked or scored Attribute Value for each attribute or the instance group in the portion of the plurality of attributes to the display interface for display.

[00068] In another aspect of this method, the at least two of the one or more instances are grouped based on one or more of: respective Attribute Value, respective number of votes received, respective attribute category and respective rank or score of reliability.

[00069] In an further aspect, a non-transient computer-readable medium containing program instructions is disclosed, the instructions, when executed by a processor, causes a computer to: a) receive, from a plurality of data sources in real-time, and store in a data storage device, a plurality of data items and a plurality of attributes for each of the plurality of data items, each attribute having one or more instances, each instance having an Attribute Value; b) receive, from a display interface, a query to display a portion of the plurality of attributes; c) receive, from the display interface, one or more votes from one or more users for one or more Attribute Values for one or more of the plurality of data items; d) determine a rank or score of reliability for each of the one or more Attribute Values based on the one or more votes received; and e) provide a highest ranked or scored Attribute Value for each attribute in the portion of the plurality of attributes to the display interface for display.

[00070] In an embodiment, a computer-network system for storing, displaying and ranking attributes relating to a plurality of real-time data items is provided, the system comprising (a) at least one module configured to display a plurality of attributes relating to a plurality of data items based on a user query; (b) at least one module configured to receive one or more votes from one or more users; (c) at least one module configured to assess and rank the reliability of the plurality of attributes for each of the plurality of data items; (d) at least one data store configured to store the plurality of attributes for each of the plurality of data items.

[00071] In another aspect the system comprises a user or data source ranking module that ranks the one or more users or data sources based on quality of contributed content of the one or more users or data sources.

[00072] In another aspect, a user or a data source can receive an appropriate ranking tier through contributing content that is ranked high by other users and other criteria such as number of views on contributed content. This ranking tier can be a factor in determining validity of contributed data.

[00073] In another aspect, attributes contributed relating to a data item are additionally ranked based on the system's perceived confidence in the attribute value independent of user voting. The system confidence in a particular attribute pertaining to a data item may be determined by many factors including, but not limited to, the history of the attribute as it relates to the data item, the plurality of attributes of the same category pertaining to the same data item contributed within a time period, and so on.

[00074] In another aspect, the reliability of an attribute pertaining to a data item is additionally ranked based on the state of the plurality of attributes of all categories pertaining to one data item. Attributes which are not consistent with the current perceived state of the data item may be ranked lower or removed from, or otherwise disabled for display in the data store. In this aspect factors including, but not limited to, the attribute relationships to attributes of different categories and to location and time data may contribute to the reliability ranking of an attribute.

[00075] In another aspect the reliability of an attribute pertaining to a data item is additionally ranked based on elapsed time since its contribution.

[00076] In this respect, before explaining at least embodiments described herein in detail, embodiments described herein are not limited in application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. Other embodiments may be practised and carried out in various ways. Phraseology and terminology employed herein are for the purpose of description and illustration and should not be regarded as limiting.

DESCRIPTION OF THE DRAWINGS

[00077] Embodiments described herein may be better understood when consideration is given to the following detailed description which makes reference to the annexed drawings wherein:

[00078] FIG. 1 is a system diagram illustrating the components of the System or Platform 200 according to some embodiments.

[00079] FIG. 2 is a sample chart illustrating expiry thresholds in different regions according to some embodiments.

[00080] FIG. 3 is a sample data model according to according to some embodiments.

[00081 ] Figs. 4 to 18 illustrate various sample user interface (Ul) screens of Mobile App 280 according to some embodiments, and particularly the WaitShare illustrative example embodiment.

[00082] FIG. 19 is an illustrative diagram which depicts how in embodiments described herein a decaying vote value might influence the Attribute Trust Score of an attribute.

[00083] Figs. 20a and 20b illustrate employing a decaying vote value with an Attribute Trust

Score threshold at zero which may allow attributes to expire or begin to expire at later or earlier times depending on the votes received according to some embodiments.

[00084] Figs. 21a and 21 b illustrate a variety of different functions which in embodiments described herein may be applied to vote values, attribute instance Attribute Trust Scores, attribute category scores or data item scores.

[00085] FIG. 22 illustrates that in embodiments described herein, vote decay functions may be adjusted in a variety of ways and by a variety of factors.

[00086] FIG. 23 illustrates how in embodiments described herein employing vote decay functions of different types for upvotes and downvotes may influence the Attribute Trust Score of an attribute.

[00087] FIG. 24 illustrates the Vote Value and Attribute Trust Score of an attribute, which has received one upvote and one downvote and employs vote decay according to some embodiments.

[00088] FIG. 25 illustrates how in embodiments described herein which employ vote decay, attributes which receive frequent upvotes at one time may be distinguished from attributes which receive upvotes regularly.

[00089] FIG. 26 is a sample chart of the submitted Attribute Value with time for a particular attribute category which illustrates a Confidence Threshold according to some embodiments. [00090] FIG. 27 illustrates a sample Confidence Factor Distribution and the associated

Confidence Threshold according to some embodiments.

[00091 ] FIG. 28 illustrates attributes with similar values that may be grouped and ranked as a group, while the displayed value may be the range within the group or top scoring value according to some embodiments.

[00092] Figs. 29b and 29b illustrate a locations radius and coordinates may be adjusted to correspond to user activity pertaining to that location according to some embodiments.

[00093] Figs. 30a and 30b illustrate a location may be further delineated by employing surrounding locations whose location radii overlap according to some embodiments.

[00094] FIG. 31 is an illustrative system diagram providing another example implementation of

System or Platform 200 according to some embodiments.

[00095] FIG. 32 is an example diagram illustrating relationships between data items, attributes, instances and attribute values according to some embodiments.

[00096] In the drawings, embodiments are illustrated by way of example. It is to be expressly understood that the description and drawings are only for the purpose of illustration and as an aid to understanding, and are not intended as a definition of the limits of various embodiments described herein.

DETAILED DESCRIPTION

[00097] Dynamic data may be in constant flux hour by hour. To date the dynamic nature of such data have acted as invisible or inaccessible variables with enormous impacts on the choices we make. The value of globally collected, constantly updated and validated data as new inputs for day to day or critical decisions is incalculable and its application to many industries and endeavours far-reaching. From Apple™'s Siri to Google Now™, validated real-time information may be the next level in real-time services or marketing.

[00098] Embodiments described herein can enable the community to manage the dynamic nature of such information by validating, in real-time or substantially in real-time, the reliability of the gathered information. Consequently, users, who may not have access to this information first-hand, are able to access reliable information on the data item they wish to query. If information is unavailable, users can be queried to add this information at an incentive if they are in a position to do so. In this way information gaps can be filled and validated by the crowd in the same way as other data sources. If information is present, users can be asked to validate information by voting on its accuracy. In this manner, every user can add value to the crowd-sourcing system by contributing in one way or another.

[00099] The present disclosure describes systems and methods of attributing reliability to the current state of fluctuating real-time data. It presents a platform for collecting, validating and displaying real-time information from dynamic data sources by employing crowd consensus and data metrics. [000100] In embodiments described herein, as shown in FIG. 1 , the Platform 200 can gather data from multiple distributed dynamic data sources 295 and determines which collected data is valid at least in part by consulting users in a Crowd Sourcing Platform 290 and employing data metrics. The Platform 200 can further gather data from the Crowd Sourcing Platform290 by means of a Display Interface 330 (see e.g. FIG. 31 , described below). Each collected attribute is sent to the Platform Validation Service 250, given an Attribute Trust Score 540, and ranked against the plurality of other collected attributes of the same category which pertain to the same data item 500. The attribute or combination of attributes with the highest Attribute Trust Score 540 is displayed or returned as the most reliable information available at that time.

[000101] In one aspect of embodiments described herein, the Platform Validation Service 250 implements a process which assigns Attribute Trust Scores 540 to individual collected attributes. The Platform Ranking Service 230 may then rank the plurality of attributes of a particular category by their Attribute Trust Score 540 at that time. The Platform Validation Service 250 may promote, discount, group or outright discard attributes 510 or attribute values 530 in an effort to produce Attribute Trust Scores 540 which best reflect the reality of the situation in real-time. The Platform Validation Service 250 may consider metadata other than the attribute value 530, including, but not limited to, the location of submission, the time of submission and the attribute data source, all of which may influence the resulting Attribute Trust Score 540.

[000102] In one aspect of embodiments described herein, the Attribute Trust Score 540 is also influenced by the Crowd Sourcing Platform 290. Votes for or against an attribute indicate that the attribute value 530 is valid or invalid at the time of the vote and also indicate an increase or decrease in the confidence ascribed to the data source by the crowd. This allows the Platform Source Ranking Service 230 to learn which data sources are trustworthy and promote or discount attributes collected accordingly. Votes by users whose metadata reveals they cannot have specific knowledge relating to the particular attribute and/or data item 500 they are voting on may be discarded or disallowed. Metadata surrounding these votes may be kept by the Platform 200 to be used as indicators of data inconsistencies at a later date, such as if the location of a data item 500 changes. In addition to votes, the Platform 200 also provides a reporting system whereby users of Crowd Sourcing Platform 290 may indicate that data is incorrect and specify why they believe so. This provides a secondary control on the validation of attributes or data items in the case where voting is insufficient. The Platform 200 may handle these reports manually or aggregate them and use the metadata to correct errors automatically.

[000103] In another aspect of embodiments described herein, the Attribute Trust Score 540 is also modified by data metrics independent of the crowd sourced voting. Over time, the Platform 200 may learn which attribute values 530 are 'likely' in a particular scenario (location, date, time-of-day, etc), and this may adjust the Attribute Trust Score 540 or cause the attribute to be discarded. Similarly the Platform 200 learns the time during which attributes of a certain category remain valid, and discards or discounts their Attribute Trust Score 540 after this time has expired. Rapid integration of these processes allows the Platform 200 to adapt to spatially or temporally localized events and prevents discarding potentially valid data. Despite these learned behaviours, the Platform 200 may prioritize the input of the crowd sourcing platform such that data from an untrustworthy source or which does not appear logical statistically may be voted valid by users with first-hand knowledge of the data item or particular attribute 510.

[000104] According to the embodiments described herein, a single data attribute 510 pertaining to a particular data item 500 may be collected from Crowd Sourcing Platform 290 or other dynamic data source, ranked by voting on the attribute 510 by Crowd Sourcing Platform 290 and sanity-checking by data metrics, and finally, if it ranks highly enough, displayed back to users of Crowd Sourcing Platform 290 or elsewhere in real-time or substantially real-time. In this way the Platform 200 allows for the most reliable and up-to-date metadata about a particular data item 500 to be displayed as dictated by community opinion.

[000105] Referring now to FIG. 1 , according to embodiments described herein, a system 200 for the Platform 200 may utilize a cloud-based service or Server Module 210 (or simply "Server 210"), which may be a Software as a Service (SaaS). Various services reside in the cloud, providing back-end support for Crowd Sourcing Platform 290. Crowd Sourcing Platform 290 may be implemented with methods such as, but not limited to, a Mobile Application or Mobile App 280, a website or Web Application 285 or combinations thereof, and so on. Mobile App 280 can be implemented on a wirelessly networked device (not shown). Mobile App 280 can conform to the device's hardware and operating system, and to push and pull content over a wireless network from the various services such as Ranking Service 230, Contribution Service 240, Validation Service 250, Report Service 255, Expiry Service 260, and Source Ranking Service 270. Mobile App 280 can be a software module, a hardware module, or a combination of hardware and software components. For example, it may be written in HTML5 codes, and optionally bundled using Apache™ Cordova to run on iOS™ or Android™ devices. The user interface or Ul can adapt to device screen size and resolution in order to provide superior user experience regardless of device type. Consumers or users may also access a website or Web Application 285 that can enable access to data via a web interface. In addition, API Service 215 may be enabled to provide validated realtime data to consumers or businesses not participating in Crowd Sourcing Platform 290.

[000106] It should be understood that the wirelessly networked device as described may consist of a hand-held two-way wireless paging computer, a wirelessly enabled palm-top computer, a mobile telephone with data messaging capabilities, a portable digital media player, or a wirelessly enabled laptop computer, but could be any type of mobile data communication device capable of sending and receiving messages via a network connection. The majority of current mobile communication device users, however, use a mobile telephone with data messaging capabilities, such as server addressing capabilities such as Short Message Service ("SMS") or Multimedia Messaging Service ("MMS") or data including GPRS or 3G or LTE. [000107] For some embodiments described herein, user contribution via the Crowd Sourcing Platform 290 may only be allowed through a method or methods (such as Mobile App 280 or Web Application 285) where certain metadata is available. For example, user proximity to a location can be a key factor in building accurate data about that location, so in this case, user contribution may be limited to Mobile App 280 where location data is readily available.

[000108] In another embodiment according to yet another aspect, user contribution via Crowd Sourcing Platform 290 may be made with all methods available, including, but not limited to, Mobile App 280 and Web Application 285.

[000109] Referring now to FIG. 31 , another exemplary system diagram for System or Platform 200 is shown. Server 210 is similar to the Server Module 210 in FIG. 1 , including one or more of API Service 215, Ranking Service 230, Contribution Service 240, Validation Service 250, Report Service 255, Expiry Service 260, Timer 265 and Source Ranking Service 270 (not shown). Server 210 may be operably connected to, or include, a Display Interface 330 that is configured to interface user devices (not shown) such as mobile phones or tablet computers, in order to collect and transmit various data. Data Store 220 may also receive data from and transmit data to Display Interface 330. In some example embodiments, Display Interface 330 may be an interface designed to communicate with Crowd Sourcing Platform 290 (including Mobile App 280 and Web Application 285, not shown), and therefore act as a Data Source on its own.

[0001 10] Both Server 210 and Data Store 220 may collect and receive data from dynamic Data Sources 295 such as various databases, user devices, hospital personnel and so on.

[0001 11] Server 210 can be configured to receive from one or more of the Data Store 220, the dynamic Data Sources 295 as well as Display Interface 330, a plurality of data items 500 and a plurality of attributes 5 0 for each of the plurality of data items, each attribute 510 having one or more instances 520, each instance 520 having an Attribute Value 530. As shown in FIG. 32, an Attribute Value 530 can be an instance of an Attribute 510. As each attribute instance 520 has a corresponding Attribute Value 530, the term "attribute instance" or "instance" 520 may be used interchangeably with the term "Attribute Value" 530 throughout this disclosure. A ranking or scoring of an attribute 510, or of Attribute Values 530, can mean a ranking or scoring of instances 520 belonging to the attribute 510, based on respective attribute value 530a, 530b, 530c for each instance 520a, 520b, 530c. In some example embodiments, the ranking or scoring of an attribute value 530a, 530b, 530c or of an instance 520a, 520b, 530c is determined or adjusted based on a respective Attribute Trust Score 540a, 540b, 540c of the attribute value 530a, 530b, 530c or the instance 520a, 520b, 530c.

[0001 12] Each Attribute Value 530 may have a corresponding Data Source 295, which may also correspond to a contributing user of Crowd Sourcing Platform 290 facilitated by Display Interface 330. The Server 210 may also receive one or more user votes from the Display Interface 330 regarding one or more instances 520 or the respective corresponding Attribute Value 530 for each instance 520. Server 210 can be further configured to determine a rank or score 540 of reliability for each of the one or more Attribute Values 530 based on the one or more votes received, and provide a highest ranked or scored Attribute Value or Instance for one or more attributes 510 to the Display Interface 330 for display.

Data Item Attributes

[0001 13] A data item 500 may have a variety of attributes 510, divided into at least two types: static and non-static attributes. A static attribute can be defined as a property or category of information relating to a specific data item 500 (e.g. a location or event) that does not generally vary on a day-to-day or weekly basis, such as a location address, the date or type of an event, a telephone number, a product number and so on. Non-static or dynamic attributes 510 may be types of information that may be accurate at time of recording or observation, but typically vary or expire over time due to changes in circumstances. For example, non-static attributes may include the number of products left in stock, the star-rating for an online review, the number of people waiting in line at a medical facility or the names of the doctors on duty there, and so on.

[0001 14] Due to the dynamic or changing nature of non-static attributes 510, the Server Module 210 may implement a server process as part of the Validation Service 250 that is operable to verify the accuracy of the non-static attributes 510 associated with a particular data item in real-time or substantially real-time.

[0001 15] Attributes 510 pertaining to a particular data item 500 may also be of a variety of categories. For instance if the data item 500 is a location, attribute categories may include, but are not limited to, the location address, the location name, the location type (business, clinic, private dwelling etc.), location contact information, current occupants, hours of operation, and so on.

[000116] In accordance with embodiments described herein, each attribute 510, static or non- static, may have more than one instances 520 contributed by various dynamic data sources 295, possibly including users or contributors from Crowd Sourcing Platform 290. For example, data source A may contribute that Clinic X has three doctors on duty, while data source B may subsequently or concurrently contribute that Clinic X has five doctors on duty. In this example, "number of doctors on duty" is the attribute 510, pertaining to data item 500 "Clinic X", while three or five (doctors) is the particular instance 520 having an attribute value 530 of the attribute 510 contributed by users. As can be appreciated, each attribute 510 may have unlimited number of instances 520 contributed by data sources or users; however, typically for each attribute 510, only one instance 520 or group of instances 520 can be accurate, or closest to being accurate, at any given time. This accurate or closest to being accurate instance or group of instances of attributes, ranked by users, may be referred to as the attribute of the data item 500 throughout this application. That is, "attribute" as used in this application may refer to a category of information (e.g. "number of doctors on duty today"), or a particular instance or group of instances (e.g. "three doctors on duty today") that is accurate or most accurate among all instances of the attribute. [0001 17] In embodiments described herein, a non-static attribute 510 of a particular data item 500 may be in the form of ranges of values or groups of instances. For example, if the top two ranked delivery times pertaining to a delivery service data item are 20 minutes and 30 minutes, the attribute 510 may display "Delivery times currently between 20 and 30 minutes". This can lessen the impact of volatile data such as number of people in line causing accurate information being deemed inaccurate by voting (via Crowd Sourcing Platform 290) as data changes quickly. In addition, an attribute 510 in the form of a range of values tends to change less frequently than a single value. A range generally cannot be applied to static attributes. An attribute 510 in the form of a range of values may be formed by Validation Service 250 by grouping instances 520 based on factors including, but not limited to, similar attribute value 530, similar Attribute Trust Score 540, attribute category and so on.

[0001 8] In an aspect of embodiments described herein, a time-sensitive expiry threshold may be associated with each of the non-static attributed 510 relating to a data item. The expiry threshold may have an initial, default value (e.g. two hours from time of collection). The expiry threshold may also be adjusted by Server Module 210, for example through Expiry Services 260, based on a variety of factors.

[000119] While static attributes 510 do not have an expiry threshold, the duration of time between an attribute instance 520 being first contributed by a user and the attribute instance 520 being downvoted may have an impact on how users are ranked by server module 210.

[000120] Referring now to FIG. 32, another exemplary diagram for data items 500 and its attributes 510 is shown. In this embodiment, a plurality of data items 500 may be stored in a Data Store 200 (not shown), each data item 500 may include a plurality of attributes 5 0, each attribute having one or more instances 520. Each instance may have a corresponding Attribute Value 530. The Attribute Value 530 may be a user-contributed value or a default value. Each instance 520 or Attribute Value 530 may further have a determined Attribute Trust Score 540 associated thereof. The Attribute Trust Score 540 of each instance 520 or each Attribute Value 530 may be used to rank or score the Attribute Values 530, instances 520, or attributes 510 of a data item 500.

[000121] Even though only three instances 520a, 520b, 520c are shown in FIG. 32, it is to be appreciated that more than three instances may exist for one or more attribute 510.

[000122] The number of instances 520 for a given attribute 510 may fluctuate over time. The value of each of the Attribute Values 530a, 530b, 530c may also fluctuate over time.

[000123] In another embodiment, instances 520 may be eliminated from the data structure of a data item 500, whereby each attribute 510 simply has one or more Attribute Values 530.

Users

[000124] In embodiments described herein, users or consumers refers to people employing Crowd Sourcing Platform 290 to access, update, validate or contribute information from the Platform 200. In another aspect, users may also refer to an organization or entity which employs validated data in the same way, or directly from the Platform 200.

[000125] In an aspect of embodiments described herein, users may perform a variety of actions such as build or update data via Contribution Service 240 or validate data items and data item attributes via Validation Service 250. Users may contribute information by adding, editing and voting on data item attributes, becoming another data source and helping to build a real-time validated Data Item Data Store 220 ("contributors"). Alternatively, users may simply view or consume the validated data provided by the Platform 200 ("consumers"). In embodiments described herein, contribution of real-time information particular to a data item 500 may only be allowed for users with first-hand knowledge of the data item 500. For example, users far from a medical facility may be prevented from contributing information that might require a user to be physically at a location, such as the number of people waiting in line.

[000126] In an aspect of embodiments described herein, users may report incorrect data items 500 or attributes 510 when voting is insufficient. In these cases users may be allowed to specify what they deem is incorrect about the attribute 510 or data item 500. For example, a user may report a duplicate data item, allowing the Platform 200 to remove it from display. In embodiments described herein, reported items or attributes may be handled offline. According to another embodiment, reported items or attributes may be resolved automatically by Server Module 210 based on the user trust-worthiness or ranking provided by Source Ranking Service 270 and the reason provided for the report. Reports on attributes 510 or data items 500 which are deemed truthful either by the Platform 200 or other mechanisms may have an impact on the ranking of the contributing data source by Source Ranking Service 270.

[000127] According to embodiments described herein, users of Crowd Sourcing Platform 290 may not be required to register or log in prior to consuming and/or contributing data. Anonymous users rank, points, trust-worthiness score and other metadata may be tracked by means of a unique token assigned by Server Module 210 as well as stored by the user in non-transient storage on the device used to access Crowd Platform 290. Server Module 210 may store this token in User Data Store 225. This unique token may be transmitted to the Server Module 210 with each request and checked against the token stored by Server Module 210 in User Data Store 225. In this way the user is accessing Server Module 210 for each request may be determined by his or her unique token. The lack of requirement of user sign-on eliminates a barrier to both using and contributing to the Platform 200, which in turn makes it more valuable to the users.

Voting

[000128] In an aspect of embodiments described herein, Crowd Sourcing Platform 290 enables users to view and vote on the 'most correct' data pertaining to a particular data item 500 which are displayed or returned by the Platform 200. Users may vote for ("upvote") or against ("downvote") attributes 510 or data items 500, and these votes may be processed by Validation Service 250. This voting process allows the 'crowd' to determine which attributes 510 are valid or invalid. An upvote can indicate that a voting user feels that the displayed data is correct or most accurate, while a downvote indicates the user feels this attribute 510 is incorrect or inaccurate. A user may only be permitted to vote the same way on an attribute 510 once. However, an upvote followed by a downvote, or vice versa, is permitted. Besides upvotes and downvotes, other suitable voting systems or mechanisms may also be used, such as "liked"/"disliked", or "agreed'V'disagreed", and so on.

[000129] In embodiments described herein, the total votes on a data item 500 or attribute 510 pertaining to a data item may be displayed back to consumers as a feedback mechanism. This may allow users to infer the finer grained degrees of accuracy of an attribute. For example, it may allow a user to distinguish between an attribute 510 which is voted highly amongst many others and an attribute 510 which is voted somewhat highly amongst few.

[000130] In another aspect of embodiments described herein, a time-sensitive expiry threshold may be applied to individual votes in addition to the attribute expiry threshold.

[000131] In yet another aspect of embodiments described herein, users who contribute attributes 510 or data items 500 may not vote specifically on their contribution. Other restrictions on user voting may also apply, such as, but not limited to, disallowing multiple votes of the same type (upvote or downvote) on a single attribute 510, or disallowing votes on a particular attribute 510 after a certain time, and so on.

Source Ranking

[000132] In embodiments described herein, data sources and/or contributing users can themselves be ranked based on their contribution, or otherwise associated with a trust-worthiness based on their contribution. Associated with such a trust-worthiness factor, the validity of information or instances 520 of attributes can be determined based on positive or negative ranking (e.g. upvote or downvote) by the more or most trusted users of Crowd Sourcing Platform 290. For example, a data source may consistently contribute data that is deemed inaccurate by the crowd. The data source can be deemed non-trusted or assigned a low value of trust-worthiness. An instance 520a of an attribute 510a that receives an upvote from a non-trusted data source may be weighed less than an instance 520b of an attribute 510b that receives an upvote from a trusted source. That is, Source Ranking Service 270 can be configured to assign a variety of degrees of trust-worthiness based on the accuracy of contributed data by each data source or contributing user.

[000133] In an aspect of embodiments described herein, the user or data source trust-worthiness is distinct from any points or ranks assigned via gaming mechanics. Data source or user trustworthiness may be assigned partially based on the upvotes or downvotes attributed to data collected from that user or data source and specifically not based on gamification points gained from contributions. The assigned trust-worthiness value may contribute positively or negatively to a user's rank or points value. In this instance points displayed to the user would be solely to encourage further participation, and the Source Ranking Service 270 would rely primarily on the vote-based trust-worthiness. [000134] In embodiments described herein, the user or data source trust-worthiness may be assigned as a score rather than a ranking. This allows two users or data sources to be scored identically, forcing accuracy determination via another avenue.

[000135] In an aspect of embodiments described herein, the trust-worthiness of data sources and users may be ranked or scored independently. This allows different weights to be applied to upvotes or downvotes which factor into the trust-worthiness score depending on the contributor type (user or data source).

[000136] In another aspect of embodiments described herein, attributes with a high volatility - i.e, have a high frequency of change or update - can be identified and given a volatility factor. The higher the volatility factor of an attribute 510, the lower the impact of receiving a downvote is on a user's ranking. This avoids penalizing users that may have contributed valid data that is only relevant for a short period of time. This volatility effect may also be applied to attributes 510 whose accuracy is contested continuously by the crowd, reducing both the negative and positive effects of votes while the attribute is so contested.

[000137] Although static attributes may not expire in the same way as non-static attributes, in an aspect of embodiments described herein, the length of time between the contribution of a static attribute and when it is deemed inaccurate by Crowd Sourcing Platform 290 still has an impact on user trustworthiness. For example, if a contributor has added an address attribute 5 0 to a location data item 500, and that attribute instance 520 is downvoted by the crowd a year later, the negative impact to that contributor's ranking is relatively small, or none, as the address associated with a business may change over time. However if the static attribute is downvoted after only one day, that contributor may be considered as having contributed poor or bad data and the server can lower the contributor's ranking and assign less weight to that data sources contributions in the future.

[000138] In another aspect of embodiments, user trust-worthiness may also be influenced by how the crowd votes compared to the user. For instance, if a user upvotes one attribute 510, while the larger user community deems this attribute 510 to be inaccurate, the user may incur a penalty to his or her trustworthiness. The penalty incurred, if any, may be dependent on the time-elapsed between when the user vote occurred and when the crowd at large determined the attribute 510 or data item 500 inaccurate.

Server Processes

The Server Module 210 may implement processes which can be used to determine which data items 500 or attributes 510 pertaining to a data item are the most accurate at that time. The processes validate and rank the plurality of attribute instances 520 and data items 500 against the plurality of other data items and attribute instances and return only the most accurate up-to-date results for a consumer query. Processes implemented by Server Module 210 may comprise several components which make up implementations or partial implementations of the Validation Service 250, Expiry Service 260, Source Ranking System 270 and Ranking Service 230. Variables to these server processes can include, but are not limited to, user or data source metadata (such as location, local time, etc.), attribute or data item upvotes and downvotes, time-elapsed since an action or event, any pertinent data or attribute interrelationships and data relationships to data source or data item metadata, and so on. Variables can be weighed by a variety of factors such as data source or user ranking, proximity to a location, time- elapsed since an upvote or downvote and so on.

Validation

[000139] According to embodiments described herein, Validation Service 250 can record each vote contributed by a user in relation to a displayed attribute or data item and calculate a total vote score for the attribute 510. This vote score may be stored with the attribute 510 in Data Item Data Store 220 and displayed to consumers. For each attribute 510 in relation to a data item 500, if there are more than one contributed instances 520 or groups of instances 520 related to the attribute 510, the instance 520 or group with the highest Total Vote Score may be given preference. In embodiments described herein the attribute instance 520 or group of attribute instances 520 with the highest Total Vote Score may be deemed the most accurate data and displayed to the consumers in relation to the data item 500.

[000140] The Total Vote Score may be defined as the total number of net positive votes N, which is the difference between the number of upvotes U and the number of downvotes D. That is, N = U - D.

[000141] In an exemplary embodiment of embodiments described herein, the Total Vote Score may be modified to include considerations of the ranking of voting users as determined by Source Ranking Service 270 to produce an Attribute Trust Score 540. The user or data source ranking may be influenced in part by a trust-worthiness score as detailed above. User trust-worthiness as determined by Source Ranking Service 270 can reflect the level of activity and/or quality of a user's contributions, with higher rankings reflecting high activity and high quality submissions. Higher ranked user's upvotes and/or downvotes can be given more weight than lower ranked users. This enables confirmed 'high quality' users to have a greater influence on the Attribute Trust Score 540 and thus which attributes are displayed. Vote weighting can also have the added benefit of preventing 'low quality' users from purposefully skewing results.

[000142] As a result, the Attribute Trust Score 540 for a particular attribute instance 520 or group of instances 520 may be determined by the equation:

* UpvoteW eighty -∑_jl_Q(DownvoteValue_j * DownvoteW 'eight _;) (1 ) where UpvoteValue represents the value of an upvote, DownvoteValue represents the value of a downvote, n represents the total number of received upvotes, m represents the total number of received downvotes, UpvoteWeight represents an associated weight of the corresponding upvote for each upvote received, and DownvoteWeight represents an associated weight of the corresponding downvote for each downvote received. The UpvoteValue and DownvoteValue represent how much value to assign each upvote before weighting. This value may be 1 for upvotes and downvotes, however this may be adjusted based on a number of variables including, but not limited to the time elapsed since the vote occurred, the grouping of attribute instances, and so on. The weights UpvoteWeight and DownvoteWeight may be associated with a user ranking of the user who has contributed the upvote or downvote. These may be influenced, as described above, by a user trust-worthiness score determined by Source Ranking Service 270.

[000143] According to the equation above, if the sum of the downvotes, taking into account the individual vote values and weights, overcomes the sum of the upvotes, then a negative Attribute Trust Score 540 may be assigned to the attribute instance 520. Similarly if the sum of the upvotes overcomes the sum of the downvotes, the attribute instance 520 may receive a positive Attribute Trust Score 540.

[000144] In an aspect of embodiments described herein, the data items, attribute instances 520 or groups of attribute instances 520 which fall below a threshold Total Vote Score or Attribute Trust Score may be removed or expired from the Data Item Data Store 220. In this way collected data which has been deemed sufficiently inaccurate by the crowd can be prevented from ever being displayed or ranked again even if their expiry time has not yet elapsed. In embodiments described herein, the threshold score may be a static value, or a dynamic value which depends on the vote history relating to a particular attribute category. In either case, the threshold may be adjusted by the system administrator at any time. In an aspect of embodiments described herein, this threshold can be applied to cases where only a single attribute instance or group of instances has been collected for a particular attribute category pertaining to a data item. Applying a threshold in this case would prevent attribute instances which have the highest Total Vote Score by default from being displayed if they fall below this threshold. In an aspect of embodiments described herein, the data items, attributes or groups of attributes may instead be removed based on a threshold Attribute Trust Score. In another aspect of embodiments described herein, the threshold for display may be defined as simply a total number of downvotes an attribute has received. For example, if the threshold is set at four downvotes, an attribute instance or group of instances has received four or greater downvotes may be removed by the Expiry Service 260 and not displayed to consumers.

[000145] In another aspect of embodiments described herein, attribute instances 520 or groups of attribute instances 520 which fall below a threshold Total Vote Score or Attribute Trust Score may be disabled for display, or disabled for ranking or scoring, in the Data Item Data Store 220. However, the expired or disabled attribute instances 520 or groups of instances 520 may be used to perform heuristic analysis or to inform historical data such as historical confident values. Attributes and attribute values (instances of attributes) may be removed from ranking or displaying features, but not from the data store. The system may still use past expired values to inform historical confidence values, and so on. [000146] In an aspect of embodiments described herein, the value of each individual vote may decay over time. In this way votes which occurred nearer to the current time are more valuable, and contribute more to the Attribute Trust Score 540.

[000147] Over time, the server process may implement machine learning techniques, such as heuristic learning or cluster analysis, which inform the validity of an attribute or data by analyzing real-time contributed data, optionally based on grouping the data by attribute category, data item or a group of data items, or other context. Context may refer to geography, language, user demographics, data item types or other groupings. In yet another aspect of embodiments described herein, an attribute's Attribute Trust Score 540 may be influenced by data metrics associated with collected data organized by these contexts. For example, the value of a submitted attribute may be compared to previous submissions in the same category over the same period of time. For example, a line length of 100 is submitted during a period of time which, historically, has low line length values (2, 5, etc.). In such a case the Attribute Trust Score 540 may be adjusted to reflect how likely this data is to be accurate according to the data history of line lengths for that data item or similar items. In this way the Attribute Trust Score 540 may be adjusted independently of the votes received or the trustworthiness of any users involved. The Attribute Trust Score 540 may be adjusted in this way according to a number of variables including, but not limited to, attribute category history, location, time of contribution, attribute category interrelationships, and so on. In an exemplary embodiment, these data metrics are used to define thresholds for attribute values 530 within the category, beyond which the Attribute Trust Score 540 is lowered significantly, or the attribute is discarded altogether. In this way outliers may be eliminated even if they come from a very trusted user or data source. It is worth noting that this effect may be considerably less important than user voting, so that an outlier value that is voted truthful by the crowd can be redeemed.

Source Ranking

[000148] As described above, data sources (including users) may be ranked by the Source Ranking Service 270. This service may employ a variety of factors to rank data sources including, but not limited to, trust-worthiness score, gamification points or ranking tier, voting history (if applicable), and so on. The data source rank of a user may then influence the UpvoteWeight and DownVoteWeight variables Equation 1 above and serve to adjust the final Total Vote Score or Attribute Trust Score 540 of an attribute 510 or data item 500.

Expiry

[000149] In general, the server processes include rules to capture that non-static attribute values 530 do not remain valid forever, and enables expiry of attribute value 530 and the respective attribute instance 520 after a certain period via Expiry Service 260 in the Server Module 210. In embodiments described herein, attribute instances 520 and attribute values 530 expired in this way are removed, or disabled in the Data Item Data Store 220 such that the attribute instances 520 or attribute values 530 are no longer accessible for ranking, voting or display. If an attribute instance 520 expires, all edits or votes associated with the attribute instance 520 are no longer in effect. If the attribute instance 530a was the top ranked attribute by Ranking Service 230, another attribute value 520b may be chosen for display after it expires, and if there are no attributes remaining to display, the attribute display may return to its default state. In other words, once the most accurate attribute instance or group of attribute instances expires, the next highest ranked attribute is displayed if available. Attributes and attribute values (instances of attributes) may be removed from ranking or displaying aspect, but not from the data store, in some embodiments. The system may still use past expired values to inform historical confidence values, and so on.

[000150] The server processes may also learn the duration over which non-static attributes 510 or data items 500 of a certain category or context remain accurate. Over time, Expiry Service 260 may adjust the expiry threshold of an attribute value based on various factors or contexts, including, but not limited to, the attribute category, the previous expiry times for attributes of that category, the time of day, the time elapsed since a certain event, geography, data item type, and so on. Downvotes are the way in which users in Crowd Sourcing Platform 290 indicate an attribute is no longer valid. In an exemplary embodiment, the process can implement machine learning techniques (such as heuristic learning or cluster analysis) to define that an attribute is valid for T minutes by collecting downvotes on that attribute and adjust the expiry threshold from its default value to T minutes. That is, user voting over time may also be used by the server process to discover expiry patterns.

[000151] In another aspect of embodiments described herein, if downvotes occur ahead of the default expiry threshold Td by T amount of time (in minutes, hours, days, or weeks) consistently in a given context, the expiry threshold can be shortened to Tn, which can be defined as Tn = default expiry threshold Td minus T. Similarly, if downvotes do not occur within the default expiry threshold Td in a given context, the expiry threshold Tn can be extended until it reaches a point where downvotes are consistently occurring, at which time Expiry Services 260 can assume it has passed the point when an attribute should expire in the given context. For example, assuming that the default expiry threshold for an attribute is 4 hours, if an instance 520 of an attribute 510 pertaining to a data item 500 is upvoted at least once during the first 2 hours since its contribution, and then receives downvotes for the final hour of the expiry threshold, the expiry timer in that context may be reset to 2 hours for all future contributed attribute instances 520 of the same category for that data item 500. If the attribute is instead upvoted for the entire 4 hours, then the timer may extend that wait time until 40 minutes. As 40 minutes approaches, the same determination may be calculated, and so on.

[000152] In another embodiment, certain attributes may be set to be valid for a specific duration depending on context. Once an instance 520 of an attribute has been entered by a contributor and set as the attribute value, an expiry threshold in the form of a default timer can begin keeping time, taking into consideration the context of the attribute. If upvotes continue to be contributed by trusted users right up to the end of the default timer, it can be safely assumed that the default expiry threshold for that context was not long enough and it can be extended. Conversely, if downvotes occur before the default expiry threshold is reached for that context, the expiry threshold is deemed too long and can be shortened for that context. This is demonstrated in FIG. 2, where it is seen that the expiry of an attribute may be adjusted per context (in this case country) by the rate of downvotes supplied by trusted users.

[000153] In embodiments described herein, the Expiry Service 260 may expire or begin a timer to expire attributes 510 only once their Attribute Trust Score 540 reaches a low threshold. For example, if this expiry trust threshold is set to zero, an attribute may remain active in the Data Item Data Store 220 until its Attribute Trust Score 540 becomes negative and/or remains negative for a period of time. In an exemplary embodiment, individual votes may also expire over time, which, when combined with the trust threshold and equation described above, forces an attribute to receive constant upvotes in order to 'survive'. In this case, the expiry may be integrated into the Attribute Trust Score 540 equation above as a time-dependent decay function applied to the vote value. This method may also be combined with the above embodiments to provide context dependent expiry thresholds. If such a method is employed, since static attributes may have a different expiry behavior, the Attribute Trust Score 540 calculation may be modified for static attributes to reflect this (generally much slower) expiry mechanism.

[000154] Once the length of the expiry threshold has been adjusted, the same validation checks may be performed on each attribute as before the adjustment. In this way the expiry threshold is constantly being adjusted in order to reflect agreement among trusted users, using real-time or substantially real-time crowd-sourced information.

Ranking

[000155] Finally the data item 500, attribute instance 520 or group of attribute instances 520 may be ranked against the plurality of data items, attribute instances or groups of instances by the Ranking Service 270 according to its Attribute Trust Score 540. As mentioned above, the Attribute or Data Item Trust Score may take input from the Expiry Service 260, Validation Service 250 and Source Ranking Service 270. In an exemplary embodiment, the top ranked data item or attribute pertaining to a data item is displayed to consumers. In another embodiment, several of the top ranked attribute instances or groups of attribute instances pertaining to a particular data item may be displayed.

[000156] In some embodiments described herein, an instance 520 or group of instances 520 of a particular attribute category may only be displayed if it has not been expired by Expiry Service 260 and is above a threshold Total Vote Score or Attribute Trust Score as mentioned above, regardless of its contributor trustworthiness or ranking against the plurality of other attribute instances or groups of instances pertaining to that data item 500. In embodiments described herein, if all attribute instances or groups of attribute instances have been eliminated in this way, no data is displayed. In another embodiment, if all attribute instances or groups of attribute instances have been expired, a default value may be displayed. [000157] In an aspect of embodiments described herein, a user may create and submit a new instance 520 of this attribute 510, however it may not be immediately displayed. Incumbent attributes which are highly voted by the crowd may be deemed more accurate by the system and prove difficult to dislodge. The submitted new instance 520 of attribute may receive upvotes equivalent to the currently displayed attribute or wait until the current attribute has expired before it may be displayed.

[000158] In embodiments described herein, the total Attribute Trust Score 540 may decline over time in some fashion. This may solve the problem above by allowing newer attributes with fewer net upvotes to possibly rank higher than older attributes with more net upvotes.

[000159] The processes implemented by Server Module 210 maintain integrity by enforcing one or more principles on data sources, such as:

• The server process weighs data source input on a series of criteria as described above.

When the Attribute Trust Score 540 for two attributes results in a tie, the most recently entered value breaks the tie.

• Users may be required to have first-hand knowledge of the data item 500 or attribute category in order to contribute or vote on certain attribute information. For example, if the data item 500 is a location, the user may be at or near the location in order to contribute or vote on non-static attributes related to the location. The restrictions required to ensure users have first-hand knowledge of the attribute may change depending on the context of the attribute 510 or data item 500, but they may include restrictions on location, time of day, user relationship to the data item and so on. In one exemplary embodiment, these restrictions may apply only to non-static attributes, depending on the context of the attribute. In another embodiment, restrictions on voting and contributing data apply to both static and non-static attributes.

• Negative voting on data contributed by a data source lessens the value of the data sources future contributions via a negative impact on the data source ranking as dictated by Source Ranking Service 270.

• The process allows for contributions from "verified" or "validated" data sources for a data item 500 to be integrated with other contributed attributes. For example, a verified hospital may publish official wait times, which may be weighed higher than data from other sources by Server Module 210, or be expired differently by Expiry Service 260. In another embodiment data contributed in this way may not be voted on, but is instead displayed as "official" to consumers. In yet another embodiment, officially contributed attribute instances by a verified contributor may be treated identically to contributed content from other sources. • Removing or deactivating attributes which have expired or passed a voting threshold as described above.

EXAMPLE IMPLEMENTATION: WAITSHARE

[000160] An excellent example of data which is currently unavailable due to unreliability is healthcare facility information. In general, real-time information such as a healthcare facility's current availability and the number of people waiting in line is not available to the public. Published wait times by the healthcare facilities may provide limited helpfulness but in too many parts of the world are unreliable or unavailable. Even directory-type information such as phone numbers, hours of operation, and addresses remain at times inaccurate, incomplete or unlisted. The need for healthcare services is often immediate, unexpected and often occurs in places unfamiliar to the patient. Real-time or substantially real-time information regarding healthcare services in these instances would be a tremendous boon to patients in need of care.

[000161] WaitShare is an implementation of the System or Platform 200 and processes described above which aims to provide accurate up-to-date medical facility location information by employing users at the medical facilities themselves. WaitShare aims to provide information such as, but not limited to, the location wait times, the location line lengths, the hours of operation, location address, name and type as well as a list of services provided, and so on.

Platform Implementation

[000162] In some embodiments described herein, WaitShare is implemented in the System 200 with a cloud based Server Module 210 and Crowd Sourcing Platform 290, employing both a Mobile App 280 and Web Application 285. Users of Crowd Sourcing Platform 290 are the primary data source, while external data sources by "verified" medical facility personnel (e.g. published or unpublished emergency room wait times) are also considered.

[000163] Each category of location data (wait times, hours of operation, etc.) may be contributed by users of Crowd Sourcing Platform 290 and are considered as static or non-static attributes categories within the Platform 200. Other attributes may include, but are not limited to, individual service listings, location open status, location type (Hospital, Clinic, Pharmacy, etc.), time closed for the day, country, website and so on.

[000164] Data Item Data Store 220 contains a series of locations and their associated data attributes. These locations may be contributed by users or gleaned from an external Application Programming Interface or from static sources elsewhere. User Data Store 225 may contain information about user actions and trust-worthiness in the context of the Platform.

[000165] In the WaitShare embodiment, users refer to people wishing to view or contribute medical facility information by employing Crowd Sourcing Platform 290. WaitShare users may not be required to register to use the Platform. As described above this allows for a low barrier to entry, where users may simply download Mobile App 280 or access Web Application 285 to employ the Platform 200. However, in order to track user contributions, votes and ranking, WaitShare makes use of unique user tokens by the Server Module 210. When a user accesses Crowd Sourcing Platform 290, a check is performed to determine if the user has stored a user token from a previous access (e.g. Mobile App 280 may accomplish this by storing the user token on the device). If a user token cannot be found, a new token is generated by Server Module 210. This token is then sent to the user to be stored locally in non-transitory storage. A new record can then be generated and stored in User Data Store 225, and this token may be associated with each of the users subsequent actions.

[000166] Alternatively, users may register a user account to view his or her action metadata, points or ranking as stored in User Data Store 225. Additionally, WaitShare provides registered users with the ability to choose and display a listing of favorite locations.

[000167] In the WaitShare embodiment, users are encouraged to contribute by means of gaming mechanics. Users are incented to contribute or update data via Contribution Service 240 or validate data items or attributes via Validation Service 250 by attributing points to users who perform such actions. For example, users as contributors may earn status or points as they contribute accurate information as judged by votes from Crowd Sourcing Platform 290. Users are ranked based on these points, which are based in part on upvotes or downvotes a user receives in relation to his or her contributed content or data. Points may be received by users for a variety of events or acts. For example, points may be awarded for:

1. Contributing content or data (e.g. data items or attributes) regarding a data item.

2. Receiving votes (upvotes or downvotes) for contributed attributes

3. Having a contributed attribute appear in search results or on a map.

4. Views or other use of contributed attribute or data item.

5. Validating attributes.

Points may also be removed if an attribute which a user upvoted is later deemed to be incorrect.

[000168] The points, ranking tier and rank amongst other users may be displayed prominently to the user near the top of the display on Mobile App 280 or Web Application 285 in order to spur participation.

[000169] In the WaitShare embodiment, different events or contributions carry different point values. Points are given to users at the time the event occurs (upvote, data/attribute viewed, etc.) and are generally not calculated retroactively. Points, ranking and other gaming mechanism data is stored along with the respective user tokens in User Data Store 225. [000170] In the WaitShare embodiment, users are enabled and encouraged to vote on wait times, line lengths and other attributes in order to validate data. As described above, these upvotes or downvotes contribute to the final ranking of an attribute instance 520 or group of attribute instances amongst the plurality of attribute instances of that category for that data item. In this case the data item 500 is the healthcare facility, and the attribute may be a type of location data such as a wait time (the category) and the upvotes or downvotes contribute to the Attribute Trust Score 540 for that submitted value of the wait time (the attribute instance 520). WaitShare may also restrict votes on non-static attributes (e.g. wait time) to those submitted by users within a certain proximity to the healthcare facility location. This ensures that users validating wait time or line length data have first-hand knowledge of the attribute 5 0s they are voting on.

[000171 ] Additionally the WaitShare embodiment may implement a Reporting Service 255. This service allows users to report problems with locations or attribute 5 0s which may be difficult to identify by voting alone. For instance, if a location is duplicated within the Data Item Data Store 220, a user may report one of the duplicate locations, allowing System 200 to review the report and act accordingly. If a report is viewed to contain accurate information which improves the accuracy of the displayed data, the reporting user may receive points. Conversely a user who contributes data which is reported by one or several verified reports may lose points or receive a warning.

[000172] WaitShare employs points gained through contributions and validation as a means to achieve different user ranking tiers. These user ranking tiers are designated as levels of user- trustworthiness and may apply as weights to user upvotes and downvotes. Ranking tiers as designated by Source Ranking Service 270 may be calculated according to a static set of pre-defined point ranges, or a percentage based system.

[000173] As a result, WaitShare calculates the Attribute Trust Score 540 according to the following Equation:

* DownvoterRank_j){2) where UpvoteValue represents the value of an upvote, DownvoteValue represents the value of a downvote, n represents the total number of received upvotes, m represents the total number of received downvotes, UpvoterRank represents a weight associated with the ranking tier of the corresponding upvoting user for each upvote received and DownvoterRank represents a weight associated with the ranking tier of the corresponding downvoting user for each downvote received. The UpvoteValue and DownvoteValue represent how much value to assign each upvote before weighting. This value may be 1 for upvotes and downvotes, however this may be adjusted based on a number of variables including, but not limited to the time elapsed since the vote occurred, the grouping of attribute instances, and so on.

[000174] WaitShare implements a context dependent set of expiry thresholds as defined above and implemented in Expiry Service 260 of the Server Module 210. These expiry thresholds may be updated either directly by the administrator or by observing the timing of user downvotes in relation to the attribute contribution time as described above. Context or category dependent attribute expiry may be achieved by implementing an attribute table in Data Item Data Store 220 which stores expiry times per attribute category, as well as points values for contributing, and so on. When an attribute 510 (or an instance 520 of that attribute 510) is contributed to System 200, it is recorded in Data Item Data Store 220 along with the time of contribution. When it comes time for Ranking Service 230 to rank the attribute 5 0 (or an instance 520 of that attribute 510), it may simply check that the time elapsed between the time of contribution and the current time has not exceeded the time stored in the attribute table for attributes of that category or context. If more time has elapsed than allowed by the attribute table, the attribute 510 (or an instance 520 of that attribute 510) is deemed expired and discarded by Ranking Service 230.

[000175] The Server Module 210 employed by the WaitShare example illustrative embodiment validates user contributed data by means of user votes. Equation 2 is used to calculate the Attribute Trust Score 540 for each instance 520 of that attribute 510, taking into account the user trust-worthiness ranks assigned by gamification point ranking tiers. This allows votes by users who contribute frequently crowd verified data to have more weight than users who view or consume data only. Since points are also given for validating data, and removed for validating inaccurate data, users who consistently validate correct data may also be more influential.

[000176] Finally, attributes 510 such as wait time or line length are ranked by Ranking Service 230 by Attribute Trust Score 540. The top attributes of each category for each health facility are displayed as the most accurate data available at that time. As a means of promoting other attribute instances, a list of the top attributes is presented to the user when editing or updating a value in an attribute category. This has the added benefit of encouraging users who may input a similar attribute value as a separate instance, to instead validate an existing attribute instance, therefore increasing its ranking.

Data Model

The Server Module 210, Mobile App 280, other components of the system 200, and corresponding workflows are supported by the data model illustrated in FIG. 3.

Workflows of Server 210 and System 200

[000177] Server Module 210 may include the following workflows:

• Search for a Location;

• Add a Location;

• Vote on a Location's Information;

• Edit a Location's Information;

• Vote on or Update a Location's Wait Time and/or Number of People Waiting; and • Submit Feedback or Report,

each of which are explained in further detail below.

Search for a Location

[000178] In the WaitShare example embodiment, when Mobile App 280 is launched on a user's mobile device, Mobile App 280 can detect the current location of the user based on the mobile device's location determination functions such as GPS and WIFI triangulation. The number of locations that have been updated in the last day, last week, or since the users last WaitShare action within a certain distance 'near' the user may be displayed.

[000179] Additionally if there are locations within a certain proximity to the user, a blue button may appear on the main screen prompting the user to "Share Now at location x" where location x is the location nearest the user of any type. When clicked, this button takes the user directly to a details page (described below) containing more information about the location, and allowing the user to being contributing or validating attributes about that location. Alternatively this button also contains a drop-down menu with a list of the y closest locations to the user, so that they may select which location the main button takes them to.

[000180] FIG. 4 illustrates a sample user interface screen of Mobile App 280 according to an aspect of embodiments described herein.

[000181 ] In one aspect of the WaitShare example embodiment, based on a user's current location, healthcare locations fitting a type (e.g. Clinic, Hospital or Pharmacy) selected by the user may be queried by a search engine. When a search is performed, the following steps can occur:

1. A search request, such as an asynchronous JSON request, is sent from the Mobile App 280 on the mobile device to the Server Module 210, invoking appropriate search routines. The request includes the user token, the location or location type selected by the user, and the user's current location information such as GPS coordinates;

2. A search of Data Item Data Store 220 is performed for locations of the proper type and within a certain proximity to the users submitted GPS coordinates. This step may be enhanced by also querying external Application Programming Interfaces which allow access to healthcare facility information (e.g. Nominatim™) or location information (e.g. OpenStreetMaps).

3. The search returns a set of search results. Each individual location result can contain a unique location identifier which may be associated with attribute instances also stored within Data Item Data Store 220. 4. The returned location data can be sorted based on the Mobile App 280's current display setting (e.g. recommended, closest, open now, and so on) and a response is returned to the Mobile App 280 for displaying to the user.

[000182] In one aspect of the WaitShare example embodiment, as seen in FIG. 5, for each location returned by the search engine and displayed to the user, the location name, address, current wait time, number of people in line, distance from the current location and open status may be displayed. By default, the locations are displayed in a 'recommended' order. The recommended sort order can be based on a process that takes into consideration of the following factors or variables: distance to the location from current location, whether the location is currently open, the current wait time, and the number of people waiting at the location. Optionally the locations in the search result can be further filtered by the services they offer.

[000183] As seen in FIG. 6, the user may also change, or filter, the type of the location being displayed in a list of locations by selecting the desired place type in a drop-down menu for types.

[000184] In addition, as seen in FIG. 7, the sort order of the locations can also be altered or filtered by selecting the drop-down menu for sorting. For example, selecting sort order "Closest" may cause Mobile App 280 to list the locations in the search result based on location proximity to the user, regardless of whether they are currently open or what the current wait time may be for each location. For another example, the sort order "Open Now" can cause Mobile App 280 to list those locations that are currently open or about to "Open Soon".

[000185] In another embodiment, a user may also view the locations in a search result on a map by selecting the Show Map option, as seen in FIG. 8. The Sever Module 210 and/or Mobile App 280 can leverage appropriate map API functionality (e.g. MapBox) to display the user's current location and proximity to each healthcare location in the area.

[000186] In yet another embodiment, a user may select any healthcare location to view additional information regarding the selected location. For example, on a location detail screen as shown in FIGs. 9a and 9b, a location name, address, and distance from the user's location are displayed. The current wait time and current number of people in line (both as contributed and voted on by users) may be displayed. In the lower portion of the screen, additional attributes of the location are displayed. For example, the additional static attributes may include name of the location, type of location, address of the location, phone number, web site, open hours for today and the next six (6) days, and/or the services offered by the location. Any attribute XXX 510 (e.g. a web site) which does not currently have a contributed instance or value can be shown with or in the form of 'Add a XXX... ' (e.g. Add a web site... ). An indicator may also be shown in the Hours section to indicate if a location has closed early for the day, as shown in the hours for FIG. 9c. Additionally an indicator may be shown near the top of the details screen which indicates the open or closed status of the location currently being viewed. [000187] Furthermore, from the location detail screen, a user can call the location by clicking the

Call Now button, which can cause the user's mobile device to location a phone call to the phone number associated with the location. If the user is on a tablet, the user can be asked if he or she would like to add the phone number to contacts for future use. In addition, if the user selects the Directions button, an appropriate map application available on the mobile device can be launched (e.g. Apple™ Map on an iPhone™ or Google™ Map on an Android™ device), providing the appropriate directions from the user's current location to the location in question.

Add a Location

[000188] As described earlier, System 200 can include a Data Item Data Store 220 that stores real-time (or substantially real-time) and validated information of all healthcare facilities or locations. Data Item Data Store 220 can be augmented by a search engine API, such as Nominatim API. If a user knows of a healthcare location that is not currently listed in the System 200, he/she may add the healthcare location to System 200 via Mobile App 280 or Web Application 285. For example, as seen in FIG. 10, the user may add a location by clicking on the "+Add" button available on the Mobile App 280 or Web Application 285. In one example implementation of the WaitShare example embodiment, the user may be within a defined proximity to the healthcare location in order to add information regarding the location to System 200. In another implementation, the user may add information regarding a location regardless of his current location.

[000189] Next, the user can proceed to choose the type of healthcare location that is to be added. The user's current location based on the mobile device's GPS and WIFI triangulation can be displayed on the map, and the surrounding geography including street names can be populated. This may allow the address and street name of the location to be automatically entered in the correct field. The location type, by default, is set to the type that was chosen by the user. The user can then fill in the appropriate information that he/she would like to regarding the location. For example, the name of the location may be a mandatory field that must be filled in, while other additional information (e.g. FIGs 11a and 11b) may be optional. The optional fields may be populated or contributed by other users (or the current user) at another time. Examples of additional information about the location can include the street number, phone number, website, hours of operation, services offered, average wait times, and so on.

[000 90] In the WaitShare example embodiment, hours of operation attributes of a location can be shown as a rolling view of the times that a location is open. The data model can support an open and close time for each day of the week. Once an open time and close time for a week day for a location have been entered and stored in Data Item Data Store 220, the database 220 can record the same open and close times for that day of the week going forward, until it is changed or overridden by a user or a system administrator. Mobile App 280 or Web App 285 can display hours of operation for a location for at least seven days, starting from the current day. Alternate hours can be recorded for a day up to a week in advance, and can include days that have hours outside of a normal schedule, such as holiday schedules. This alternative schedule feature can be useful on holidays or for locations such as clinics, which tend to have more flexible hours.

[000191] As described the process of adding a location is seamless. Many attributes 510 can be added during this process, or later by the same contributor who initially added the location, or others who may wish to earn points to increase their ranking tier or status. There is no requirement to login to add a location, as points can be tied to a device or to a user account if one is created.

[000192] A contributor can receive a comparably high number of points for adding a location. Early adopters can drive up their online ranking tier or status fairly quickly by adding accurate missing locations. Alternatively, a location that is added and quickly downvoted can drive down a user's ranking tier.

Vote on a Location's Information

[000193] While a user is viewing a location's information, he/she can vote on each attribute 510 of information regarding the location as shown on Mobile App 280 (see e.g. FIG. 12) or Web Application 285. For example, the following static or non-static attributes 510 of a location may be voted on: name, location type, address, phone number, website, hours of operation, average wait times, line lengths and so on. The user can click thumbs up (upvote) to agree with an individual attribute value 530, or thumbs down (downvote) to disagree with an individual attribute value 530. The total number of upvotes or downvotes from the crowd of users, taking into account their respective ranking tiers (and thus trustworthiness) can subsequently determine the Attribute Trust Score 540, which, when ranked by Ranking Service 230, determines whether an attribute 510 can be displayed or can be considered accurate or inaccurate.

Edit or Submit a Location's Information

[000194] In addition to voting on an attribute information, in the WaitShare example embodiment, a user may also 'edit' or rather, submit a different set of information relating to an attribute 510, or to add information that is not currently displayed. That is, users do not actually 'edit' displayed attribute in the sense that the edited content would replace the existing attribute value 530 right away. A current version of the attribute instance 520 can be on display until the expiry threshold or the timer runs out. While a user can create and submit a new instance of this attribute if he or she chooses, the submitted new instance of attribute needs to receive upvotes equivalent to the currently displayed attribute or wait until the current attribute has expired before his or her newly contributed attribute may be displayed. That is, an attribute with a high Attribute Trust Score 540 as voted by users of the Crowd Sourcing Platform 290 may not be easily changed or edited, as the existing information may still be considered accurate from the perspective of the system.

[000195] In the WaitShare example embodiment, a user can start by selecting an editing or contributing icon, such as a pencil icon (e.g. see the pencil icon next to "Add a website" in FIG. 12), for the particular attribute information to be edited. Next, another user interface (Ul) screen pops up (e.g. FIGs. 13a-13e), where the user can enter a new piece of information for a particular attribute 510. Alternately, the user may upvote or downvote (e.g. FIG. 13d) a previously entered piece of attribute information. As previously described, the attribute information (or instance) 520 with the highest Attribute Trust Score 540 at any given time can be recorded as the most accurate information and displayed as an attribute for a location. The Attribute Trust Score 540 is defined by Equation 2 as defined above.

[000196] If the pencil icon relating to hours of operation of a location is clicked on by a user, a screen can open where the user can edit the hours of operation for the location (see FIG. 14a). The opening or closing time for each day can be edited by selecting from a drop-down menu. The same hours as edited can also be applied to all of days of the week. Additionally, the hours of operation may be set persistently for all days of the week or a single day (e.g. Wednesdays). A 24 hour and Closed setting are also provided, as well as an option to indicate the location has closed early for that day only. Such an alternative closing time may be stored in Data Item Data Store 220 only until the end of the day according to the local time of the contributing data source. A user may also thumbs up (upvote) or thumbs down (downvote) a previous hours entry. As previously described, the hours of operation information with the highest Attribute Trust Score 540 at any given time can be recorded as the most accurate information and displayed as an attribute for a location.

[000197] In another example embodiment, as seen at FIG. 14b, a user can add information regarding services offered by the location by clicking on the "+" button. The user can also remove existing services offered by clicking on the "X" next to a displayed service. The user can then be asked to confirm that he or she wants to remove the service.

Vote on or Update a Location's Wait Time and/or Number of People Waiting

[000198] When a user is physically at a healthcare facility or location, he/she may vote on the wait time and the number of people waiting in line using the thumbs up and thumbs down buttons (see example shown at FIG. 15a). In addition, the user can edit the wait time and number of people waiting in line by clicking on each of these buttons individually.

[000199] If the wait time attribute is selected, another screen can pop up where the user can select a new current wait time for the location from an hours drop-down menu and a minutes drop-down menu (see e.g. FIG. 15b). Alternately or concurrently, the user may upvote or downvote a previous wait time. The wait time with the highest Attribute Trust Score 540 at any given time can be recorded as the most accurate information and displayed as an attribute 510 for a location.

[000200] Number of people in line may also be added, edited, or voted on in a similar manner, see for example FIG. 16. Submit Feedback or Report Items

[000201] At any point in the application, the user may submit feedback on the application (see e.g. FIG. 17). When the 'Submit Feedback' button is selected, a screen opens where the user may provide their email address (in case additional follow-up is necessary), subject of the email, and a feedback message. If the user is registered, or has previously provided an email to the Feedback dialog, their email address can be pre-populated, leaving only the subject matter to enter.

[000202] Additionally users who navigate to the details screen of a particular location and find some inconsistency or inaccuracy which is outside the realm of normal voting (e.g. two location entries for the same physical medical facility), a user may select the "Problem?" button. This button opens a dialog (FIG. 18a) whereby users may specify a reason for the problem at the location from a predefined list as well as enter relevant comments similar to the Feedback box (FIG. 18b). The predefined list of options may include, but is not limited to, wrong information, permanently closed, duplicate location, wrong location, offensive information, other, and so on. A user may then select "Send" which sends the user token, user GPS coordinates, report type, time of day, location identifier and many other pieces of user and location information to Server Module 210. As detailed above, the Server Module 210 may deal with these reports either programmatically or in an offline manner. Users who submit reports which prove valid upon review, either by Server Module 210 or a server administrator, may receive points. Conversely, users whose data is reported on as inaccurate for whatever reason may also lose points, subject to time- elapsed, context and type of data.

ALTERNATE PLATFORM IMPLEMENTATIONS

[000203] In some embodiments described herein, one or more alternative implementations exist for different parts of the Platform 200, which are described below.

Vote Decay

[000204] Information or attribute values 530 contained by attribute instances 520 or groups of attribute instances 520 is accurate for a particular time and place. Accuracy is determined by aggregating upvotes and downvotes from the Crowd Sourcing Platform 290, then later expired by Expiry Service 260. However the nature of dynamic data is that it is constantly in flux, so an attribute which a user has upvoted at one time may no longer be valid at a future time. As a result, an upvote on an attribute far in the past may no longer represent a reasonable measure of its current validity.

[000205] In embodiments described herein, this effect is mitigated in part by scaling or decaying the value of each individual vote as time increases from its time of contribution. This allows votes which happen nearer to the current time to have a higher value, while those contributed further in the past, representing votes on the 'past accuracy' of the attribute, are less impactful. In order to calculate the Attribute Trust Score 540 as defined by Equation 1 the vote value of each vote is summed. In the case of individual vote decay, each vote may contribute less value to this total as time increases from its time of contribution, which causes the Attribute Trust Score 540 to be constantly moving towards zero with time. This effect may allow newer attribute instances with fewer votes to outrank older attributes or groups of attribute instances which have many upvotes, but no longer represent reality. In this case the votes which contributed to the older attribute may have less value in general than any votes on the newly contributed attribute, which may allow its Attribute Trust Score 540 to overtake the older attributes.

[000206] In another exemplary embodiment, the vote value decay is paired with a threshold Attribute Trust Score at which attribute instances or groups of attribute instances are expired. In another aspect of embodiments described herein, a timer may begin at the time the threshold Attribute Trust Score is reached. If the attribute remains at or below the threshold for the duration of the timer, it is expired. An example of a logical timer value may be the same amount of time it takes an attribute of a particular category or context to decay to the Attribute Trust Score threshold upon receiving one upvote. In another aspect of embodiments described herein, a definitive minimum Attribute Trust Score is set (e.g. -20) in addition to the timer previously described such that attributes which reach this Score are expired regardless. In this way, since vote decay causes Attribute Trust Scores to trend towards zero, attributes whose Attribute Trust Scores do not remain high enough may be expired by Expiry Service 260. This forces an attribute to receive constant upvotes in order to 'survive'. Not expiring the attributes immediately upon reaching the threshold serves to allow attributes which are volatile or 'contested' (receive nearly equal upvotes or downvotes spread out over time) to persist if desired.

[000207] Additionally this Attribute Trust Score threshold expiry (e.g. zero) results in attribute instances expiring dynamically depending on the timing and value of votes received. As the Attribute Trust Score 540 for an attribute instance 520 may be dependent on the sum of the votes received by that attribute instance 520 (Equation 1 , etc), this dynamic expiry can be demonstrated by the following example. An instance 520 of an attribute 510 which receives a single upvote, may have an Attribute Trust Score 540 which begins at a maximum vote value and decays to the Attribute Trust Score threshold in an amount of time determined by the Expiry Service 260 (FIG. 19). If the attribute instance 520 receives another upvote before it reaches the Attribute Trust Score threshold, its Attribute Trust Score 540 may be increased, allowing more time before it decays to the Attribute Trust Score threshold (e.g. zero), thus increasing the expiry time (see FIG. 20a). Alternatively, if the attribute receives a downvote before it has decayed to the Attribute Trust Score threshold, the Attribute Trust Score 540 may decrease, often below the Attribute Trust Score threshold, causing it to expire or start the timer to expire immediately (Fig. 20b). This expiry information can be fed back into Expiry Service 260 as described above, to inform the expiry of all future upvotes or downvotes in that context.

[000208] Employing both the vote decay and Attribute Trust Score threshold allows us to employ these dynamic expiries by making a simple modification to the Attribute Trust Score equation defined in Equation 1. The Attribute Trust Score may then be calculated as:

where Upvote ValueMax represents the maximum value of a particular upvote (typically 1 ), DownvoteValueMax represents the maximum value of a downvote, n represents the total number of received upvotes, m represents the total number of received downvotes, UpvoteWeight represents an associated weight of the corresponding upvote for each upvote received, and DownvoteWeight represents an associated weight of the corresponding downvote for each downvote received. The UpvoteValueMax and DownvoteValueMax are typically 1 , however this may be adjusted based on a number of variables including, but not limited to the time elapsed since the vote occurred, the grouping of attribute instances, and so on. The weights UpvoteWeight and DownvoteWeight may be associated with a user ranking of the user who has contributed the upvote or downvote. These may be influenced, as described above, by a user trust-worthiness score determined by Source Ranking Service 270. UpvoteDecayFunction(time) and DownVoteDecayFunction(time) represent the normalized time- dependent decay functions applied to the maximum vote value. For a quadratic decay function, it may appear as:

DecayFunction(time)_quadratic = ~{ time_now - time_contributed)² + 1) where time contributed is the time of the vote contribution, and time now is the current time according to Validation Service 250 or Ranking Service 230.

[000209] In another aspect of embodiments described herein, the mechanism by which votes decay may be varied significantly. There are many different types of mathematical functions available which may serve to decay the vote value in different ways including, but not limited to, linear decay, rectangular decay, exponential decay, quadratic decay, quartic decay, gaussian decay, lorentzian decay and so on (see FIGs. 21 a and 21 b). Each of these decay mechanisms may also be adjusted in a multitude of ways which may allow effects such as, but not limited to, an increase or decrease in the time until the threshold Attribute Trust Score is reached, increase or decrease the maximum vote value, and so on (FIG. 22). Depending on the mathematical decay function chosen, the suite of adjustments available may be further enhanced. For example, gaussian or lorenztian decay functions may enable the maximum vote value and time until the Attribute Trust Score is reached to be linked, such that a relationship may be defined based on relevant criteria if so desired. In this manner the profile of the vote value over the time it is valid may be very finely tuned.

[000210] In another aspect of embodiments described herein, the vote value profile may be adjusted based on a number of factors, including, but not limited to, user trust-worthiness or ranking tier, context dependent expiry thresholds, if it is an upvote or downvote, and so on. For instance, it may be desirable to have an adjusted decay profile for downvotes, so that they have more or less influence on the final Attribute Trust Score (as seen in FIG. 23). Alternatively a user with a high ranking tier or trustworthiness score may employ an entirely different decay profile (e.g. a quartic or gaussian profile) which may increase his or her influence on the Attribute Trust Score. Individual users may also have different profiles for upvotes and downvotes which scale depending on their ranking tier or user-trustworthiness.

[000211] In embodiments described herein, adjusting the vote value profile also allows for each vote to have its own time until expiry. This effect allows the expiry of each vote to take into account context such as, but not limited to, geography, user ranking and/or trust-worthiness, time of day, attribute volatility, and so on. For example, due to a decrease in the number of users using Crowd Sourcing Platform 290 in a small country, each vote may be allowed to last longer, extending the life of contributed attributes in that country in general. In another example, for attributes which fluctuate more frequently, upvotes or downvotes may be allowed to expire more quickly. Changing the expiry of a vote based on context is a simple matter of adjusting the parameters of the vote decay function as shown in Equation 3.

[000212] In an aspect of embodiments described herein, decaying votes in this way also allows the Attribute Trust Score to adjust significantly with the frequency of incoming votes. For example, an attribute which receives three upvotes within a short period of time may initially achieve an Attribute Trust Score of nearly the maximum value of all three votes added together (see e.g. FIG. 25). Conversely, an attribute whose three upvotes are spaced farther apart in time may not achieve the maximum sum of the three upvote values. However since the votes all begin to decay at different times, the Attribute Trust Score of this second case may last longer before reaching the Attribute Trust Score threshold than the first case. This is advantageous because attributes with a high frequency of upvotes may climb quickly in the rankings, but may require further upvotes to 'sustain' said ranking (see again FIG. 25). This accounts for situations where a particular attribute is valid temporarily, allowing the temporary value to climb to the top of the rankings, then decay quickly, where a more regularly upvoted attribute instance may regain the top rank.

[000213] Vote decay also allows a more fine-grained view of the accuracy of a particular attribute instance 520 as voted by the crowd. For instance, if an attribute is upvoted once, then downvoted once, the Attribute Trust Score may have a small negative value since the downvote occurred second and the upvote may have decayed. The reverse is also true, which allows us to distinguish situations where an upvote occurs before a downvote etc. from the perspective of the Attribute Trust Score and Ranking Service 230 (FIG. 24). In an Attribute Trust Score system without vote decay or some other mechanism of vote time-dependence, the Attribute Trust Score in this situation would be simply zero, with no explicit way to determine the voting order. Additionally, vote decay allows for older 'entrenched' attributes with many upvotes to lose Attribute Trust Score (and thus ranking) over time, allowing newer, potentially more accurate attributes with less upvotes simply by virtue of their relative lifetimes, to rise in ranking faster.

[000214] In another aspect of embodiments described herein, vote decay may be employed without an Attribute Trust Score threshold, and be simply a means of adjusting the Attribute Trust Score for ranking purposes. In this case, attributes would not expire due to vote decay, but instead by other mechanisms employed by Expiry Service 260, such as an Attribute Trust Score-independent dynamic expiry, as described above.

[000215] In yet another aspect of embodiments described herein, the total Attribute Trust Score may be decayed instead of individual votes. This would have the advantage of significantly simplifying Attribute Trust Score calculations, but at the cost of some of the per-vote controls described above.

[000216] In this way employing a vote decay and/or Attribute Trust Score expiry threshold may allow for fine-grained control of attribute expiry, dynamic expiry timers, accounting for vote frequency, independent user or vote Attribute Trust Score customization, and so on.

Data Value Confidence

[000217] Aspects of embodiments described herein may be focused on attributing validity to information collected from static or dynamic data sources by means of upvotes or downvotes received from Crowd Sourcing Platform 290. However in some cases the contributed data, or data voted valid by the crowd may be of questionable accuracy. For example, if a user submits a wait time of ten thousand hours for a healthcare facility, this attribute instance is nonsensical under normal circumstances and should be discarded regardless of user trust-worthiness score or ranking tier or even possibly the upvotes received. With this in mind it is logical to attempt to assess the 'likelihood' of a contributed attribute 510, attribute instance 520, group of attribute instances 520 or data item 500 in addition to validation from the Crowd Sourcing Platform 290.

[000218] In embodiments described herein, System 200 implements a Confidence Factor which defines a 'most likely value' and threshold values at some deviation from this likely value for an attribute instance 520 of a particular category. These threshold values may represent either a certain level of confidence in the attribute instance value 530 (such as a 20% confidence level), which may be managed or interpreted differently by Server Module 210 or a value at which the attribute is to be simply discarded as an outlier (see FIG. 26). In an aspect of embodiments described herein, the Confidence Factor may scale or reduce the effective Attribute Trust Score 540 of an attribute instance 520 based on its deviation from the most likely value. The Confidence Factor of an attribute may depend on a number of factors, including, but not limited to, attribute location, data source location, time-elapsed since a relevant event, upvotes or downvotes, the attribute category, attribute category interrelationships, the attribute category history, relationships of attributes to other data, consistency of attribute value 530 with other attributes for that data item 500, the attribute value 530 itself, and so on.

[000219] In an aspect of embodiments described herein, the Confidence Factor may define reasonable static thresholds for an attribute of a certain category, and discard or discount the Attribute Trust Score 540 of attributes 510 or attribute instances 520 beyond these thresholds. In an aspect of embodiments described herein, static thresholds are again defined, but the Attribute Trust Score 540 of attribute instances 520 may be reduced linearly or in some other fashion the more they deviate from the most likely value.

[000220] In an aspect of embodiments described herein, the Confidence Factor is secondary to crowd voting on an attribute. That is, an attribute instance 520 which has been deemed an outlier by the Confidence mechanisms described herein may still receive a high Attribute Trust Score 540 if it is validated by many users. However an outlier such as this may require more upvotes than a non-outlier attribute instance to be highly scored by Ranking System 230. In this way outliers may be 'redeemed' based on users with first-hand knowledge of the attribute 510 or data item 500, and what is Confidence thresholds can be adjusted accordingly.

[000221] In an aspect of embodiments described herein, a Confidence Factor may be defined as a Distribution of some kind (e.g. Normal Distribution) about central value where an attribute of that category has been deemed most likely (see FIG. 27). Single or multiple Confidence thresholds may then be assigned at different deviations from the center of this Confidence Distribution. For example, outliers may be discarded beyond two standard deviations from the most likely, while attributes falling outside one standard deviation may have their Attribute Trust Scores reduced. In embodiments described herein, the value of the normalized Distribution function (fluctuating between 0 and 1 ) may be applied as a scale to the Attribute Trust Score 540 of the attribute instance 520.

[000222] In an aspect of embodiments described herein, thresholds defined by a Confidence Distribution may be dependent on the center ('most likely value') of the Distribution. The most likely value at a given time may be determined by a number of factors, including, but not limited to, the history of attributes in that category for that data item, attribute context, the last attribute value deemed accurate by the system, the ranking of contributed attribute instances and their value, and so on. For instance, the Confidence Distribution may be centered about a historical average of contributed values for that attribute category pertaining to a particular data item 500. This average value may also be weighted by the rank achieved by or Attribute Trust Score achieved by each attribute contributing to the average. In this way the path of the average value through time follows the best values as voted by the community. For instance if a new attribute is contributed and upvoted highly, this may 'pull' the average towards the value of this new contribution, so that the Confidence Distribution is centered nearer to that value when the next attribute is considered. In this case the average would be determined on an ongoing basis for each new attribute, however the most likely value may also be determined by averaging over different, more relevant periods of time, such as days of the week or months.

[000223] The Confidence thresholds may also depend on the width of the Confidence Distribution. The width of the Distribution may also be determined by a number of factors such as, but not limited to, historical values of the distribution width, the spread of values with positive Attribute Trust Scores at that time, consistency with attributes of other categories or from other data items, attribute context such as location, and so on. For instance, the width of the Confidence Distribution and thus the confidence thresholds may be reduced if at a particular time many attribute instances are being submitted near to the current center of the Distribution. This would allow for confidence thresholds to be adjusted based purely on the number and variance of submitted attribute values, regardless of votes, which may represent another measure of crowd validation.

[000224] As mentioned above, in an aspect of embodiments described herein, Confidence Factor thresholds and most likely value may be adjusted continuously with time or recalculated for each submitted attribute. Alternatively, confidence thresholds or most likely value may be calculated for a particular time period (e. g. Mondays) and reset or recalculated once this period has passed. How often the Confidence Factor thresholds are calculated may also depend on the rate at which attributes are being submitted. For example, a minimum number of attributes may be required to calculate a statistically significant Confidence threshold, such that a Confidence threshold may be recalculated very often in times when contribution rate is high, or seldom when contribution rate is low.

[000225] In an aspect of embodiments described herein, during periods where an attribute category for a particular data item receives little or no contributions, Confidence thresholds may expand. This may account for an increased uncertainty in what is likely for a particular attribute category. For example, an attribute whose last contribution was 10 minutes ago is less likely to have changed value significantly than an attribute whose last contribution was one hour ago. Expanding the confidence thresholds and/or Confidence Distribution width in these situations may prevent discounting or discarding new potentially valid information.

[000226] Confidence thresholds may also be individual to each submitted attribute. This may allow further differentiation of attributes based on the metadata of the contributing user. For instance a wider Confidence Distribution may be applied to user submissions whose location is within a certain proximity to the data item 500.

[000227] In an example aspect of embodiments described herein, the Confidence Factor is applied to the Attribute Trust Score by means of a multiplicative coefficient, producing a new Attribute Trust Score equation:

AttributeTrustScore = ConfidenceF 'actor -(AttributeValue) * AttributeTrustScoreEquation(4) where AttributteTrustScoreEquation may be one of Equation's 1 , 2 or 3 as described above depending on implementation, and ConfidenceFactor(AttributeValue) is the value of the Confidence Factor or Distribution for the AttributeValue of the attribute in question. In embodiments described herein, the ConfidenceFactor is a function which may equal a value between 0 and 1 and is dependent on the difference between the AttributeValue of a particular attribute instance and the most likely value as determined by other methods described above. [000228] In an aspect of embodiments described herein, the Confidence Factor and thresholds may be influenced by attribute category data relationships, or relationships between the attribute category and other metadata or external data. For instance, if an attribute value 530 of wait time attribute instance 520 is contributed as 3 hours and the number of people waiting in line was previously reported and validated as 33, then we may assign a lower confidence value to the contributed wait time based on a known relationship (wait times are longer with more people in line) between these attributes.

[000229] In another aspect of embodiments described herein, Confidence thresholds may be defined in part by checking the consistency of attributes with the 'current valid state' of an attribute or data item at that time. For a particular type of data items, System 200 may define rules which dictate relationships between attribute categories and define what makes a set of contributed attributes in different categories consistent with one another. These rules may allow the System 200 to judge if a contributed attribute is consistent with other attributes which have been voted accurate by the crowd at that time. For example, if an attribute category consists of a weather report on a location data item, if one user submits that it is snowing in one location while all locations surrounding his or her location within a certain proximity and time period have reported temperatures above 20 degrees Celsius and no precipitation, this new attribute may be deemed inconsistent by the Platform 200. If an attribute is not consistent with the 'current valid state', its Confidence Factor may be discounted, or it may simply be discarded.

[000230] In embodiments described herein, if a user continuously contributes attribute instances which are deemed to be low Confidence, this may impact his or her trust-worthiness score or ranking tier. In addition, users who contribute attribute instances which appear to be inconsistent or unlikely, may also lose status. For instance if the same user had contributed a wait time of 3 hours and line length of 33 at nearly the same time, it may be logical to discount the confidence (and thus possibly the Attribute Trust Score as described above) of one or both of these attributes as well as reduce the user's trust-worthiness score accordingly. This effect may be temporary or continue on an ongoing basis based on user contributions. Similarly outlier values which are not upvoted significantly may negatively affect the user- trustworthiness even if contributed by users with a high trust-worthiness score or ranking tier.

[000231] In an exemplary embodiment, in addition to the methods listed above, Confidence may be determined or influenced by a variety of statistical methods not listed here which employ variables such as, but not limited to, attribute value history per category or context, attribute value interrelationships, user, attribute or data item metadata and so on.

Attribute Instance Processing

[000232] In another embodiment, the individual attribute instances 520 submitted by data sources may be separated from what is displayed to Platform consumers. Separating displayed values from attributes in this way allows for the displayed value to instead be calculated from a group or groups of attributes, modified by the 'current system state' or otherwise processed in some way by the Platform 200 before display.

[000233] In an aspect of embodiments described herein, displayed values may be shown as a simple average of the submitted attribute instances 520 in a particular range. This average may also be weighted by data source trust-worthiness score or ranking tier or by the previous history of the attribute category.

[000234] In another aspect of embodiments described herein, separating the displayed value from attribute instances may allow relationships between attribute categories to generate estimated data. This enables System 200 to fill in gaps in data which are not populated by users. For instance, the number of people in waiting in line at a location and the wait time may be related linearly. If a user contributes both a line length and wait time attribute at the same time, a wait-per-person value may be calculated. If a user later submits only a line length or wait time, the other value may be estimated based on the calculated wait-per-person. In this case, instead of displaying a line length and the default 'no-value' for the wait time, the system may display the estimated wait time instead. Additionally, higher derivatives may be employed (e.g. the rate of change of the wait-per-person) to enhance estimations, or predict future values or Confidence thresholds.

[000235] In another aspect of embodiments described herein, displayed values may be the result of calculations performed on groups of attribute instances 520 (and their respective attribute value 530) in a particular category pertaining to a data item 500. Similar attribute instances (that is, instances with similar attribute values 530) may be grouped and displayed or ranked as one attribute instance 520. For example, if ten users submit an attribute with the same value, these may be combined into one attribute with a higher Attribute Trust Score 540. Subsequent submissions of the same attribute value 530 in that category may be treated as an upvote for, or otherwise increase the ranking of, the attribute group. In another aspect of embodiments described herein, attributes with values which fall within a certain range may be grouped and ranked as one (see FIG. 28). In this case the value displayed may be a range of values from the lowest to the highest attribute value 530 in the grouping. The Attribute Trust Score 540 attributed to the group may be the Attribute Trust Score 540 of the highest ranked instance 520 in the group, a combination or sum of the Scores in the group or a Score relative to the resulting Scores of other groups. Continued submissions to this attribute group may expand or contract the displayed range based on the Attribute Trust Score of the contributions to the group, the Confidence Factor of values in this range at this time, and so on.

[000236] In another aspect of embodiments described herein, attributes 510 may be grouped based on their value as well as Attribute Trust Score 540. For instance, a contributed attribute with a similar value to attributes in a particular attribute group may not be included if it has an Attribute Trust Score 540 below a certain threshold or cutoff. This threshold may be dependent on many factors including, but not limited to, the attribute value 530 with the highest or lowest Attribute Trust Score 540 in the group, the variance of attribute instance value 530 within the group, and so on.

[000237] In yet another aspect of embodiments described herein, attributes grouped as described above may display the average or highest scoring value in the group. Attributes contributing to a group may also increase the Attribute Trust Score of a central or highest scoring value. In this way continued contribution to an attribute group may cause the chosen value in the group to be ranked much higher. This may also cause the criteria for grouping the attribute values to change, such that as the average or highest scoring value reaches higher scores or ranks, the deviation from this value at which attributes are accepted into the group decreases.

[000238] In embodiments described herein, attributes may be grouped by means of mathematical modelling functions which provide a best-fit to a particular set of attribute instance scores (see FIG. 28). Parameters resulting from this modelling (e.g. distribution widths) may be used to define displayed ranges or Attribute Trust Score cutoff thresholds.

[000239] If attribute instances are separated from the value displayed by the Platform 200, users in Crowd Platform 290 are no longer voting on individual attribute instances, but instead the displayed value. In order to maintain the integrity of the Attribute Trust Score for each attribute, the value of the votes received may be divided up between any attribute instances used to calculate the displayed value. This vote value may apply to the Up/DownvoteValueMax in Equation 3 or similar items in Equations 2 and 1. In this way a group of three attributes contributing to a displayed value all receive a boost to their score. The value received by each attribute may be weighted according many factors, including, but not limited to, their contribution to the final displayed value, data source trust-worthiness score or ranking tier, Attribute Trust Score relative to the other attributes in the group, Confidence Factor, and so on.

Data Source Metadata Requirement

[000240] When collecting data from a variety of dynamic data sources, the metadata associated with each data item 500 or attribute 510 is particularly significant. Metadata such as local time or location collected in this way serves to connect pieces of data from different sources to form a better data set.

[000241] With this in mind, embodiments described herein may enforce or otherwise ensure that data sources employed by the Platform 200 supply valid metadata of a variety of types. This may be achieved in the case of user data sources by denying access to contributing data to the Platform 200 if certain metadata is not submitted as well. Data sources may also be required to validate that their metadata is accurate on a one-time or recurring basis.

[000242] In an alternative embodiment, data sources may not be required to submit metadata in order to employ the Platform 200 in any way. Location Based Data Adjustment

[000243] In particular, location data is useful in determining information about the world in realtime.

[000244] In embodiments described herein, data items which define or depend on the definition of a 'location' within a certain proximity or radius to particular coordinates may leverage metadata from Crowd Platform 290 to adjust the location data of a location dynamically. For instance, users who attempt to perform actions pertaining to a data item associated with a location may submit their GPS coordinates with each action (see FIGs. 29a and 29b). The locations collected in this way may be used to adjust the coordinates or proximity radius of the location they are performing actions on. If users performing actions on a location are clustered in one area (as in FIG. 29a), the radius defining that location may be reduced and its coordinates adjusted (FIG. 29b). Alternatively if the users are spread out beyond the current proximity radius, it may be expanded. Statistical methods may be applied to the cluster of user locations. For example, if the user distance from the location coordinates is x, then the standard deviation of x may define a proximity radius for that location. This is based on the idea that users may attempt to perform actions on a location when they are near it, so actions used in this way may be restricted to actions normally allowed only within a proximity to the location. Adjusting the radius or coordinates of a location in this way allows more accurate location data for a data item. It also aids considerably in determining which locations users are 'at' in an area dense with other locations. In cases such as this, the coordinates and radii of nearby locations can be adjusted until they do not overlap or achieve a low rate of user error.

[000245] In another aspect of embodiments described herein, locations which are in close proximity may further limit the definition of their 'location' by means of other nearby locations. In this aspect, locations are defined by a central set of coordinates, and a radius which defines the area of a location. However if circles formed by the radii of two adjacent locations overlap, it may be difficult to determine which location a particular user, device or data source resides in (FIG. 30a). In this case we may employ the line between the center of the locations and in particular the bisection of that line, to define new boundaries for adjacent locations. For instance, moving from a location A along the line which joins its center to a location B, it is logical that the area belonging to the location behind (A) may end at midpoint of that line rather than at the full radius of its proximity circle (FIG. 30b). This 'boxing in' of locations can be particularly beneficial when one location is surrounded by many others. In such cases location borders may no longer be defined by proximity radii, but instead by a provincial division of the area by the surroundings.

[000246] In an alternative aspect of embodiments described herein, the entire area drawn out by the intersection of the two proximity radii may be excised from the area belonging to both locations. If many locations overlap, the area where all of the locations overlap may also be removed from the areas of all locations. This allows the establishment of 'neutral ground' which may help to further define the locations in question. [000247] In another aspect of embodiments described herein, if the dimensions of a particular location are well known, the location may be given a weight, such that if it is known to be large, if bisection is performed as described above, the division line between the two locations may be moved further from the 'large' location, rather than stay at the midpoint. The reverse may also be true for 'small' locations. This may allow for locations which are near but are of significantly different sizes to adjust their respective areas accordingly.

[000248] In yet another aspect of embodiments described herein, location radii may be further adjusted if the central coordinates of other locations fall within their radii. In embodiments described herein, radii may be reduced such that they do not include the central coordinates of nearby locations.

[000249] In another aspect of embodiments described herein, attributes which depend on time in the proximity of a location or radius may be gathered from location data. For example, if a user or data source enters the radius of a location, this information may be relayed to System 200 by Mobile App 280 or other means, which may start a timer. When the user leaves the location, or performs an action outside the location radius, this timer may stop. The value of the time elapsed may be significant to the value of certain attributes, such as the wait time at a healthcare facility. This data may also be useful to businesses who wish to know how long a user spends at a location following an event or upon receiving certain information.

Additional Data Source Trust-worthiness Metrics

[000250] In addition to depending on voting by users, the Platform 200 may allow data source trust-worthiness or ranking to be influenced by other factors. The methods described below may be employed in addition to the vote-based mechanism described above.

[000251] In embodiments described herein, the time between the contribution of an attribute and its expiry (i.e. the attribute 'lifetime') may be significant to the data source trust-worthiness. If we compare the average lifetime of an attribute of a certain category to the lifetime of a particular attribute of that category, we this may indicate that this attribute was valid for a longer or shorter period than average. This may indicate the user who submitted this attribute was more or less trustworthy. Additionally the lifetime of the attribute in a category at a particular rank (e.g. it lasted 10 minutes at rank 1 ) compared to the average of a particular attribute category may be significant.

[000252] In another aspect of embodiments described herein, the data source trust-worthiness may be indicated or impacted by the sum of the areas under the Attribute Trust Score curve of an attribute during its lifetime.

[000253] In another aspect of embodiments described herein, networks of users or data sources may be employed or constructed to enhance determination of data source trust-worthiness. This mechanism is based on the idea that users may trust certain data sources more than others. A user may be allowed to vote on or favorite a data source or other user to specify that the user trusts or distrusts this data source in particular. This may impact the data source trust-worthiness for that data source for the whole community or for the voting user alone. That is, a user A may vote that a user B is trustworthy to them, and subsequently the user-trustworthiness score of user B according to user A is higher. In an aspect of embodiments described herein, a user C who has not voted user B as trustworthy may not view B as more trustworthy as before. In another aspect of embodiments described herein, user C does see an increase in user B's trust-worthiness score after a vote by user A. Furthermore, users may be allowed to specify that they trust those users who are trusted by another user. For instance user A may specify that they trust all users trusted by user B, and experience the effects of increased user-trustworthiness from these users accordingly. In general, an increase in user-trustworthiness may increase the Attribute Trust Score of attributes submitted by a user, and also increase the significance of their upvotes and downvotes as detailed above. In such a way networks of trust are formed within the data sources that consume and contribute to the Platform 200. These trust networks may be prepopulated from existing information about user relationships such as social networks or other data.

[000254] In an aspect of embodiments described herein, user investment may be considered as a measure of data source trust-worthiness. In general, users who consume and contribute to the Platform 200 frequently may be determined to be dependent on or invested in the Platform 200. Such users or data sources have an interest in keeping the data provided by the Platform 200 valid at all times, and thus may have their trust-worthiness score increased. Determination of investment may be made by a number of criteria, including, but not limited to, contribution rate, vote history, consumption rate, and so on.

Determination of Quantifiable Information from Context

[000255] In embodiments described herein, in addition to the attributes and metadata submitted by data sources. The content or value of attribute instances or data items may allow gleaning of additional information. For example, if an online review submitted for a restaurant is written in a particular language, the Platform 200 may determine that the data source originates from a particular country or region. In this way quantifiable information is obtained through analysis of the attribute value 530 or combinations of attribute values in a particular context.

Alternative User Validation Mechanisms

[000256] In embodiments described herein, in addition to upvotes and downvotes, user validation may be collected or derived from other user data. In an aspect of embodiments described herein, video or audio submitted by the user or the user's device through Crowd Sourcing Platform 290 may be used to determine validation. In an aspect of embodiments described herein, data accessible to the device used to access Crowd Sourcing Platform 290 (e.g. gyroscope data, accelerometer data, video or still image data, etc), which is relevant to a particular data item 500 or attribute 510, may be submitted to Crowd Sourcing Platform 290 manually by the user or automatically by the device once a user has been deemed to have first-hand knowledge of the attribute 510 or data item 500 in question. This may allow automatic validation of attribute values 530 in categories where data is readily available to the users device. In another aspect of embodiments described herein, how user or device data submitted in this way may not be apparent based on the raw data (e.g. video, audio, etc.). In this case a decision may be made about such contributed data to determine if it constitutes a positive or negative validation of the attribute 510. This may be achieved by processing done by Server Module 210.

APPLICATIONS

[000257] Embodiments described herein present a Platform for the collection and validation of real-time data collected from dynamic data sources and as such, may have a variety of applications. Generally speaking, any type of data or data source may benefit from an implementation of or enhanced by the addition of input from the VeroSource Platform. In particular, data which fluctuates in value frequently or which comes in general from sources of questionable reliability may be improved by the validation described above. Additionally dynamic data of a sensitive or critical nature such as real-time disaster reporting or severe weather warnings may benefit. Furthermore the validation methods presented above represent a new technology which may encourage or enable other technologies and/or applications of the Platform 200 in the future.

Healthcare Facility Information

[000258] As demonstrated by the example implementation above (WaitShare), a first application of System 200 is in collecting and validating real-time healthcare location data by employing users physically located at the healthcare locations.

[000259] Despite the ubiquity of location information available online, Hospital, Clinic and other related health facilities' location information is under represented. The healthcare industry is rightfully concerned, because this may prevent patients from seeking or finding the help they need when they need it. Access to real-time updated data about healthcare facility wait times, address, phone number and other information may be crucial in aiding patients find the medical attention they require. In this regard System 200 implementation of the VeroSource Platform may fill a real need for collecting, validating and displaying real-time medical facility data.

[000260] Initial users of this application of System 200 may include, but are not limited to, Moms (especially with toddlers), anyone without a family physician, new residents in an area, urban dwellers in a large center or high density area, travellers unfamiliar with the area, those travelling with a health condition, frequent fliers, and so on. These users share a common interest in accessing the best service in the shortest period of time. That shared interest in today's culture may drive individuals to interact through the system 200 as both a consumer and a contributor.

[000261] However, with the increasing dependence on walk-in clinics for the delivery of healthcare services one demographic, mothers with children under the age of 6, represents a large and ubiquitous group who are among the highest users of these services, and are aware of their and their children's vulnerability to inaccurate information. According to 2011 census data they are found in virtually every neighborhood in every community, and are estimated to number 30 million plus in Western Europe and North America alone, with a current birth rate in those regions that may sustain the demographic at a relatively constant size. Globally this demographic reaches into the hundreds of millions.

[000262] The example solution provides new validation technology for crowd sourced real-time data that can thereby take advantage of the example target group's (e.g. mothers with young children who may be likely to visit social networking sites) characteristics of ubiquitous distribution, high service- access requirements to local facilities, high social media adoption, and strong ethos of contributing to help fellow local group members. In turn, knock-on benefits are provided to everyone seeking access to the initial target offering (local healthcare services) at the moment in time they are needed.

[000263] Collection and validation of such medical facility information and the data and metadata provided by users of System 200 in relation to such data may be of interest to many different industries.

[000264] A list of industry participants or partners may include, without limitation, for instance:

• International or National Organizations:

• Major Pharmacy Chains

HMO's

• Pharmaceutical Companies

• Health Products Companies

• Airline & Travel Industry

• Card Companies with services to travelers

• Regional Organizations:

• Hospital Corps

• Regional Pharmacy Chains

• Local Service Providers especially Large Urban Centers:

• Clinics

• Dentists

• Doctors

• Other specialists

• Other local professionals and health service providers

Financial Trading

[000265] The financial industry employs numerous processes in day to day market trading. In today's fast paced climate, these processes are relied upon to deliver accurate and real-time predictions about every market activity. In cases where such processes make mistakes, the recovery process is long and painful. [000266] In embodiments described herein, System 200 may be used in addition to one or many financial processes to add a human factor to predictions. Day traders may vote on dynamic data coming from the financial processes which may allow the process to correct itself instead of producing an error, or to inform future decisions, learning from the collective wisdom of seasoned financial traders.

[000267] In another aspect of embodiments described herein, System 200 may be used to view or vote on the opinions of financial analysts around the world about a particular market activity. Financial analysts may be viewed as sources of data and their opinions aggregated into a single prediction or judgement which may be used to make financial decisions by Platform consumers.

[000268] Industry partners may include, but are not limited to, banks, hedge funds, financial trading institutions and so on.

Online Reviews and Comments

[000269] When deciding where to eat or what to buy, many people turn to the opinions of others for guidance. The internet provides a convenient mechanism to communicate with people with first-hand experience with the item under consideration. This need for word of mouth validation for a particular product, location or digital media has spawned online review sites such as Yelp™, Yahoo™ Answers and RottenTomatoes™. User comments and reviews are ubiquitous on the internet; whether browsing products on Amazon™, videos on YouTube™ or apps on Google Play™ or Apple iTunes™, user comments and reviews have become an integral part of how users perceive a particular item.

[000270] Unfortunately user reviews or comments often come from unknown or unreliable sources, and users looking to inform a decision based on these comments may only rely on the word of the contributing user that what they say is accurate. Furthermore, it has come to light in recent years that the industry at large may be manipulating these online reviews to bolster the image of their products or services. For instance, online restaurant directory Yelp™ is believed to have manipulated its reviews in the past, and successfully defended its right to do so in a 2014 legal proceeding. In addition, the corporate practice known as 'Astroturfing' has also been exposed to consumers. This practice entails an organization paying for positive comments or reviews for a product or service which are deliberately designed to appear to come from regular un-paid commenters. All of the above may result in a near complete loss of faith in online comments as a source of reliable information.

[000271] In embodiments described herein, System 200 may be employed to take user reviews or comments as a source of dynamic data. Users of Crowd Sourcing Platform 290 may then validate a particular review, comment or user based on first-hand knowledge of the item in question. When a user queries the Platform 200 or service employing the Platform 200 for comments or reviews about a particular item, a set of results may be returned as ranked by an Attribute or Data Item Trust Score as described above. Thus users of a service employing the Platform 200 may see the best comments or reviews as voted by the community. [000272] While many online services allow users to vote on or otherwise rate comments and reviews, the rankings provided are often simple and static in nature and do not represent the community view of a review or comment in real time. In fact many excellent reviews or comments may remain unseen due to a 'low score' on the service in question. In general, services such as the above would benefit greatly from a more sophisticated approach to ranking such as that provided by the Platform 200.

[000273] In an aspect of embodiments described herein, users providing reviews or comments may also be ranked or scored across as many websites or online services which employ the Platform 200. The more services which employ the Platform 200 the better the ranking of contributing users may become as their reviews or comments across multiple subjects and services may be analyzed and voted on for accuracy. The implementation of user tracking for users in Crowd Sourcing Platform 290 as described above may be applicable to reduce abuse by users planted by the industry.

[000274] In another aspect of embodiments described herein, users of the service which employs the Platform 200 may view reviews or comments ranked solely according to users in Crowd Sourcing Platform 290 which use this selfsame service.

[000275] In another aspect of embodiments described herein, users of the service which employs the Platform 200 may designate or otherwise promote validating users of Crowd Sourcing Platform 290 or particular commenting users as preferred. In this way networks of preferred reviewers may be created. These preferred commenters or reviewers or validating users may be assigned preference when ranking results to display to the user who prefers them, or to the community at large.

[000276] In an aspect of embodiments described herein, users of Crowd Sourcing Platform 290 which validate reviews or comments may be rewarded by gamification mechanics with rewards specific to the services or products whose reviews they validate. This provides direct feedback to the service in question about what users enjoy or dislike about their service or product.

[000277] Due to the ubiquitous nature of online reviews and comments, industry partners in this area may include nearly any online service which has users.

Other Applications

[000278] Many other applications exist for embodiments described herein. A few additional examples are provided herein for purposes of illustration..

[000279] In embodiments described herein, the Platform 200 may be used to collect and validate real-time information about the events on front line or elsewhere in a military engagement. Soldiers and military officers may update information about their surroundings either manually or by means of digital recording equipment, and this information may be aggregated by the Platform 200 and validated by other troops nearby or in other locations on the battlefield. This may provide a protocol for dealing with situations where an event transpires in real time and little information is available to individual soldiers. [000280] In another aspect of embodiments described herein, the Platform 200 may be employed to collect and validate information about an event as it transpires. Current news information may be disseminated amongst users of Crowd Sourcing Platform 290 or combined with other data provided by news media or analytics agencies.

[000281] Generally speaking any content or data that may benefit from distributed creation or management may be created or managed using System or Platform 200 according to some embodiments. Multiple parties may share and update credit profiles, medical or corporate records, and so on.

Industry Participation

[000282] Although not applicable to all applications, reward opportunities derived from user points and/or ranking tiers may also be of interest to a diverse range of industries associated with the reward market.

[000283] Integrity of the embodiments described herein may be maintained by the Server Module 210. As described above, it may display one or many attributes or data items pertaining to a particular query, with information ranked by fellow users who likely have had first-hand experiences with the attribute or data item. Specific service providers and/or organizations cannot purchase a highly ranked position or otherwise dominate the displayed results.

[000284] An organization may have the option to associate their logo or tag additional information and offers to the information returned to a query, but it cannot influence ranking generated from user voting or other mechanisms implemented by Ranking Service 230 or Server Module 210.

[000285] Organizations may further enhance the validation of real-time information by extending offers to contributors with a high trust-worthiness score or based on points used for gaming mechanisms.

[000286] Users who are ranked highly against other users, or who have a high ranking tier may be rewarded with bonus items, coupons and/or gift cards from advertisers. These rewards can be given to contributors with high point/rankings as a reward for providing good and reliable data. When the contributor redeems a bonus item, advertisers would be made aware that their investment has paid off in the form of a customer in their facility.

[000287] Among the data items 500 or attributes 510 displayed in response to a query, some may have logos, trademarks or brand names. Seeing a familiar logo (even if the associated data item 500 or attribute 510 is not the highest ranked option) may be reassuring to a user in an unfamiliar situation. Such a logo, trademark or brand name may be associated with an organization with which the user has previously had a relationship at home, or simply may be a trusted name.

[000288] In embodiments described herein, the system or platform can provide value to advertisers. For example, an advertiser may be charged to show a logo in association with a data item 500, to only a selected group of users, such as pre-qualified customers. As a query by a user for a certain type of data item or category of attribute is only presented with search results limited to the that specific type, an advertiser is able to reach a target audience that is more likely to actually consume services or products advertised.

[000289] In another embodiment, advertisers are also able to assess their return on investment. System 200 may be operable to perform analytics and generate reports to an advertiser on a continuous basis on a variety of parameters. For example, System 200 may provide:

• report on what data items or attributes users (anonymous) have viewed, when and where viewing occurred, and if they viewed, accessed or otherwise interacted with the advertiser's logo; or

• in the event that an advertiser has a promotional offer associated with a data item or attribute, a report on if and when the user collected or redeemed the promotional offer. This can be a form of instant feedback of direct sales relationship;

[000290] It should be understood that various other methods can result from the operation of the platform or system according to embodiments described herein, with its various features and utilities, as described in this disclosure.

[000291] A system in accordance with an embodiment may be implemented using a distributed and networked computing environment comprising at least one computing device.

(i) Mobile Implementations

[000292] Further enhancements may be provided wherein one or more of the computing devices are mobile devices or wirelessly networked devices, for example as illustrated in FIG. 1. For example, the network may be or include a wireless network, the wireless network including a wireless gateway for linking the wireless network to the Internet. The network-connected devices as previously described may consist of wirelessly networked operable to access the Internet via a wireless gateway (not shown). The wirelessly networked devices described may include a browser for interacting with the web server to access functions of the system (200). The wirelessly networked device may include a mobile application (280), which may include one or more utilities or features providing the record completion functions that may enable crowd users to interoperate with the system, and specifically the various services or modules so as to for example contribute data using the wirelessly networked device. In another aspect, the mobile application (280) may be configured to enable real time or near real time administrative functions, so as to enable for example modifications to or voting of data. The wirelessly networked devices could also be equipped with additional functionality for providing information regarding users, including for example a GPS receiver operable to provide GPS location information to invite particular users to complete tasks or sub-tasks or to allocate tasks to particular participating users. The wirelessly networked devices may also include one or more accelerometers or other movement sensors operable to provide movement-based or gesture-based information. [000293] It should be understood that the wirelessly networked device as described may consist of a hand-held two-way wireless paging computer, a wirelessly enabled palm-top computer, a mobile telephone with data messaging capabilities, a portable digital media player, or a wirelessly enabled laptop computer, but could be any type of mobile data communication device capable of sending and receiving messages via a network connection. The majority of current mobile communication device users, however, use a mobile telephone with data messaging capabilities, such as server addressing capabilities such as Short Message Service ("SMS") or Multimedia Messaging Service ("MMS") or data including GPRS or 3G. The present system and method therefore provides means for providing the functionality described herein, from mobile communication devices that are relatively common and inexpensive.

(ii) Example Implementation

[000294] The present system and method may be practiced in various embodiments. A suitably configured computer device, and associated communications networks, devices, software and firmware may provide a platform for enabling one or more embodiments as described above. By way of example, an example computer device that may include a central processing unit ("CPU") connected to a non- transitory storage unit and to a random access memory. The CPU may process an operating system, application program, and data. The operating system, application program, and data may be stored in storage unit and loaded into memory, as may be required. Computer device may further include a graphics processing unit (GPU) which is operatively connected to CPU and to memory to offload intensive image processing calculations from CPU and run these calculations in parallel with CPU. An operator may interact with the computer device using a video display connected by a video interface, and various input/output devices such as a keyboard, mouse, and disk drive or solid state drive connected by an I/O interface. The mouse may be configured to control movement of a cursor in the video display, and to operate various graphical user interface (GUI) controls appearing in the video display 108 with a mouse button. The disk drive or solid state drive may be configured to accept non-transitory computer readable media. The computer device may form part of a network via a network interface, allowing the computer device to communicate with other suitably configured data processing systems (not shown). One or more different types of sensors may be used to receive input from various sources.

[000295] The present system and method may be practiced on virtually any manner of computer device including a desktop computer, laptop computer, tablet computer or wireless handheld. The present system and method may also be implemented as a computer-readable/useable medium that includes computer program code to enable one or more computer devices to implement each of the various process steps in a method in accordance with the present system and method. It is understood that the terms computer-readable medium or computer useable medium comprises one or more of any type of physical embodiment of the program code. In particular, the computer-readable/useable medium can comprise program code embodied on one or more portable storage articles of manufacture (e.g. an optical disc, a magnetic disk, a tape, etc.), on one or more data storage portioned of a computing device, such as memory associated with a computer and/or a storage system.

[000296] While the above description provides examples of one or more embodiments of invention, it will be appreciated that numerous other embodiments may be within the scope defined by the following claims.

Claims

A computer-network system for collecting, validating and displaying information of a plurality of data items, comprising:

a data storage device for collecting and storing, from a plurality of data sources in realtime, a plurality of data items and a plurality of attributes for each of the plurality of data items, each attribute having one or more instances, each instance having an Attribute Value;

a display interface to display on a display device a portion of the plurality of attributes based on a query; and

a server implemented by at least one processor, the server configured to:

receive, from the display interface, one or more votes from one or more users for one or more attributes of the portion of the plurality of attributes; determine a rank or score of reliability for each of the one or more attributes based on the one or more votes received, the one or more instances, and the Attribute Values; and

provide a highest ranked or scored attribute to the display interface for display.

The computer-network system of claim 1 , wherein each Attribute Value has a corresponding data source of the plurality of data sources, wherein the server is configured to determine a rank or score of reliability for each of the plurality of data sources, and determine the rank or score of reliability for each of the one or more attributes based on the rank or score of reliability for the corresponding data source.

The computer-network system of claim 1 or claim 2, the server configured to receive, from the display interface, one or more votes from one or more users for one or more Attribute Values, determine a rank or score of reliability for each of the one or more Attribute Values based on the one or more votes received, and provide a highest ranked or scored Attribute Value of each attribute to the display interface for display.

The computer-network system of claim 3, wherein the server is configured to determine the rank or score of reliability for an Attribute Value of the one or more Attribute Values based on a rank or score of reliability associated with another Attribute Value of an instance of the same attribute.

The computer-network system of claim 3 or claim 4, the server configured to determine the rank or score of reliability for each of the one or more attributes based on the rank or score of reliability for each of the one or more Attribute Values of each instance of the respective attribute.

6. The computer-network system of any one of claims 1 to 5, wherein the server is configured to determine a rank or score of reliability for each of the one or more users based on indicia of quality of contributed content or votes of the respective user, and determine the rank or score of reliability for each of the one or more attributes by weighting any vote received from the respective user based on the rank or score of reliability for the respective user.

7. The computer-network system of any one of claims 1 to 6, wherein the server is configured to implement machine learning techniques to determine, for at least one of the one or more attributes, a system confidence value independent of voting, and determine the rank or score of reliability for the at least one attribute based on the system confidence value.

8. The computer-network system of claim 3, wherein the server is configured to implement machine learning techniques to determine, for at least one of the one or more Attribute Values, a system confidence value independent of voting, and determine the rank or score of reliability for the at least one Attribute Value based on the system confidence value.

9. The computer-network system of any one of claims 1 to 8, wherein the server is configured to determine the rank or score of reliability for each of the one or more attributes by comparing the one or more Attribute Values for each instance of the respective attribute.

10. The computer-network system of any one of claims 1 to 9, wherein each of the one or more attributes is associated with a category, and wherein the server is configured to determine a rank or score of reliability for each of the one or more attributes based on a rank or score of reliability associated with another attribute of a same or similar category.

11. The computer-network system of any one of claims 1 to 10, wherein an attribute of the one or more attributes is associated with an elapsed time since its collection, and wherein the server is configured to determine a rank or score of reliability for the attribute based on the elapsed time since its collection.

12. The computer-network system of claim 3 or claim 8, wherein an Attribute Value of an instance of an attribute of the one or more attributes is associated with an elapsed time since its collection, and wherein the server is configured to determine a rank or score of reliability for the Attribute Value based on the elapsed time since its collection.

13. The computer-network system of any one of claims 1 to 12, wherein each vote is associated with a decay value, wherein the server is configured to determine the rank or score of reliability for each of the one or more attributes by weighting the respective vote with the associated decay value.

14. The computer-network system of any one of claims 1 to 13, each of the one or more attributes is associated with metadata comprising a data source, a location of collection and a time of collection, wherein the server is configured to determine the rank or score of reliability for each of the one or more attributes based on one or more of the data source, the location of collection and the time of collection.

15. The computer-network system of claim 14, wherein the location of collection has a positive impact on the rank or score of reliability for each of the one or more attributes if the location of collection is within a predetermined proximity to a location of the data item associated with the respective attribute.

16. The computer-network system of claim 3, each of the one or more Attribute Values is associated with metadata comprising a data source, a location of collection and a time of collection, wherein the server is configured to determine the rank or score of reliability for each of the one or more Attribute Values based on one or more of the data source, the location of collection and the time of collection.

17. The computer-network system of claim 16, wherein the location of collection has a positive impact on the rank of score of reliability for each of the one or more Attribute Values if the location of collection is within a predetermined proximity to a location of the data item associated with the respective attribute for the respective Attribute Value.

18. The computer-network system of any one of claims 1 to 17, wherein the server is configured to determine a rank or score of reliability for each of the plurality of data items based on the rank or score of reliability for each of the plurality of attributes for the respective data item.

9. The computer-network system of any one of claims 1 to 18, wherein each vote comprises a Vote Value, wherein the server is configured to determine or adjust the rank or score of reliability for each of the attributes based on the Vote Value.

20. The computer-network system of any one of claims 1 to 19, wherein at least a portion of the one or more Attribute Values are dynamic fluctuating Attribute Values.

21. The computer-network system of any one of claims 1 to 20, wherein each attribute of the one or more attributes is associated with a data item and a category, wherein the server is configured to determine the rank or score of reliability for each of the one or more attributes in relation to other attributes associated with the same data item and the same category.

22. The computer-network system of any one of claims 1 to 21 , wherein the server is configured to implement machine learning techniques to determine, for at least one attribute of the one or more attributes, one or more predicted values for the at least one attribute and determine the rank or score of reliability for the at least one attribute by comparing its Attribute Value to the one or more predicted values.

23. The computer-network system of any one of claims 1 to 22, wherein the server is configured to implement machine learning techniques to determine, for at least one attribute, one or more predicted Attribute Values and determine the rank or score of reliability for the at least one attribute by comparing the one or more Attribute Values of each of its instances to the one or more predicted Attribute Values.

24. The computer-network system of any one of claims 1 to 23, wherein at least one Attribute Value is associated with a time expiry threshold value, and wherein the data storage device removes, or otherwise disables for display or ranking, the Attribute Value after expiration of time expiry threshold value.

25. The computer-network system of any one of claims 1 to 24, wherein at least one Attribute Value is associated with a time expiry threshold value, the at least one Attribute Value being an instance of at least one attribute, and wherein the server is configured to determine the rank or score of reliability for each of the at least one attributes based on the time expiry threshold value of the at least one Attribute Value.

26. The computer-network system of any one of claims 1 to 25, wherein, for each of the one or more users, the server is configured to track the respective user using a unique assigned token.

27. The computer-network system of any one of claims 1 to 26, wherein at least one vote for an attribute is associated with a time expiry threshold value, and wherein the server is configured to determine the rank or score of reliability for the respective attribute by weighting the vote based on the time expiry threshold value.

28. The computer-network system of any one of claims 1 to 27, wherein each attribute is associated with a volatility factor, and wherein the server is configured to determine the rank or score of reliability for each of the at least one attributes based on the volatility factor.

29. The computer-network system of any one of claims 1 to 28, wherein the rank or score of reliability for each of the one or more attributes comprises an Attribute Trust Score.

30. The computer-network system of claim 29, wherein each vote is either an up vote or a down vote, wherein each vote is associated with a weight, and wherein the server is configured to compute the Attribute Trust Score by a difference of a sum of each up vote weighted by its associated weight and a sum of each down vote weighted by its associated weight.

31. The computer-network system of claim 3, wherein the rank or score of reliability for each of the one or more Attribute Values comprises an individual Attribute Trust Score.

32. The computer-network system of claim 31 , wherein each vote is either an up vote or a down vote, and each vote is associated with a weight, and wherein the server is configured to compute the individual Attribute Trust Score by a difference of a sum of each up vote weighted by its associated weight and a sum of each down vote weighted by its associated weight.

33. The computer-network system of any one of claims 1 to 32, wherein the data storage device is configured to remove, or otherwise disable for display or ranking, an attribute if its associated rank or score of reliability is below a threshold value.

34. The computer-network system of claim 3, wherein the data storage device is configured to remove, or otherwise disable for display or ranking, an Attribute Value if its associated rank or score of reliability is below a threshold value.

35. The computer-network system of any one of claims 1 to 34, wherein each user or data source is associated with a trustworthiness factor based on a rank or score of reliability for the user or the data source, gamification points for the user or the data source, and historical data for the user or the data source.

36. The computer-network system of claim 3, wherein at least two of the one or more instances are grouped into an instance group, and determining the rank or score of reliability for each of the one or more Attribute Values comprises determining a rank or score of reliability for the instance group.

37. The computer-network system of claim 36, wherein the at least two of the one or more instances are grouped based on one or more of: respective Attribute Value, respective number of votes received, respective attribute category and respective rank or score of reliability.

38. A computer-network-implemented method for collecting, validating and displaying information of a plurality of data items, the method comprising:

receiving from a plurality of data sources in real-time, and storing in a data storage device, a plurality of data items and a plurality of attributes for each of the plurality of data items, each attribute having one or more instances, each instance having an Attribute Value;

receiving from a display interface, a query to display a portion of the plurality of attributes; receiving, from the display interface, one or more votes from one or more users for one or more Attribute Values for one or more of the plurality of data items;

determining, by a server implemented by a processor, a rank or score of reliability for each of the one or more Attribute Values based on the one or more votes received; and providing, by the server, a highest ranked or scored Attribute Value for each attribute in the portion of the plurality of attributes to the display interface for display.

39. The method of claim 38, wherein each Attribute Value has a corresponding data source of the plurality of data sources, and the method further comprises determining a rank or score of reliability for each of the plurality of data sources, and determining the rank or score of reliability for each of the one or more Attribute Values based on the rank or score of reliability for the corresponding data source.

40. The method of claim 38 or claim 39, further comprising determining the rank or score of reliability for an Attribute Value of the one or more Attribute Values based on a rank or score of reliability associated with another Attribute Value of an instance of the same attribute.

41. The method of any one of claims 38 to 40, further comprising determining a rank or score of reliability for each of the one or more users based on indicia of quality of contributed content or votes of the respective user, and determining the rank or score of reliability for each of the one or more Attribute Values by weighting any vote received from the respective user based on the rank or score of reliability for the respective user.

42. The method of any one of claims 38 to 41 , further comprising determining implementing machine learning techniques to determine, for at least one of the one or more Attribute Values, a system confidence value independent of voting, and determine the rank or score of reliability for the at least one Attribute Value based on the system confidence value.

43. The method of any one of claims 38 to 42, wherein an Attribute Value of an instance of an attribute of the one or more attributes is associated with an elapsed time since its collection, and the method further comprises determining a rank or score of reliability for the Attribute Value based on the elapsed time since its collection.

44. The method of any one of claims 38 to 43, wherein each vote is associated with a decay value, and the method further comprises determining the rank or score of reliability for each of the one or more Attribute Values by weighting the respective vote with the associated decay value.

45. The method of any one of claims 38 to 44, each of the one or more Attribute Values is associated with metadata comprising a data source, a location of collection and a time of collection, and the method further comprises determining the rank or score of reliability for each of the one or more Attribute Values based on one or more of the data source, the location of collection and the time of collection.

46. The method of claim 45, wherein the location of collection has a positive impact on the rank or score of reliability for each of the one or more Attribute Values if the location of collection is within a predetermined proximity to a location of the data item associated with the respective attribute for the respective Attribute Value.

47. The method of any one of claims 38 to 46, further comprising determining a rank or score of reliability for each of the plurality of data items based on the rank or score of reliability for each of the one or more Attribute Values for each attribute of the respective data item.

48. The method of any one of claims 38 to 47, wherein each vote comprises a Vote Value, and the method further comprises determining or adjusting the rank or score of reliability for each of the one or more Attribute Values based on the Vote Value.

49. The method of any one of claims 38 to 48, wherein at least a portion of the one or more Attribute Values are dynamic fluctuating Attribute Values.

50. The method of any one of claims 38 to 49, wherein each attribute of the one or more Attribute Values is associated with a data item and a category, and the method further comprises determining the rank or score of reliability for each of the one or more Attribute Values in relation to other Attribute Values associated with the same data item and the same category.

51. The method of any one of claims 38 to 50, further comprising implementing machine learning techniques to determine, for at least one attribute, one or more predicted Attribute Values, and determining the rank or score of reliability for the at least one Attribute Value by comparing the one or more Attribute Values of each of its instances to the one or more predicted Attribute Values.

52. The method of any one of claims 38 to 51 , wherein at least one Attribute Value is associated with a time expiry threshold value, and wherein the data storage device removes, or otherwise disables for display or ranking, the Attribute Value after expiration of time expiry threshold value.

53. The method any one of claims 38 to 52, wherein at least one Attribute Value is associated with a time expiry threshold value, and the method further comprises determining the rank or score of reliability for each of the Attribute Value based on the time expiry threshold value of the at least one Attribute Value.

54. The method of any one of claims 38 to 53, further comprising tracking, for each of the one or more users, the respective user using a unique assigned token.

55. The method of any one of claims 38 to 54, wherein at least one vote for an Attribute Value is associated with a time expiry threshold value, and the method further comprises determining the rank or score of reliability for the respective Attribute Value by weighting the vote based on the time expiry threshold value.

56. The method of any one of claims 38 to 55, wherein each attribute is associated with a volatility factor, and the method further comprises determining the rank or score of reliability for each of the Attribute Value based on the volatility factor associated with the attribute for the Attribute Value.

57. The method any one of claims 38 to 56, wherein the rank or score of reliability for each of the one or more Attribute Values comprises an individual Attribute Trust Score.

58. The method of claim 57, wherein each vote is either an up vote or a down vote, and each vote is associated with a weight, and method further comprises computing the individual Attribute Trust Score by a difference of a sum of each up vote weighted by its associated weight and a sum of each down vote weighted by its associated weight.

59. The method of any one of claims 38 to 58, further comprising removing from, or otherwise disabling an Attribute Value for display or ranking in the data storage device if the associated rank or score of reliability of the Attribute Value is below a threshold value.

60. The method of any one of claims 38 to 59, wherein each user or data source is associated with a trustworthiness factor based on a rank or score of reliability for the user or the data source, gamification points for the user or the data source, and historical data for the user or the data source.

61. The method of any one of claims 38 to 60, wherein at least two of the one or more instances are grouped into an instance group, and determining the rank or score of reliability for each of the one or more Attribute Values comprises determining a rank or score of reliability for the instance group.

62. The method of claim 61 , wherein the at least two of the one or more instances are grouped based on one or more of: respective Attribute Value, respective number of votes received, respective attribute category and respective rank or score of reliability.

63. A computer-network-implemented method for collecting, validating and displaying information of a plurality of data items, the method comprising:

grouping, by a server implemented by a processor, at least two of the one or more instances into an instance group;

determining, by the server, a rank or score of reliability for each of the one or more Attribute Values and the instance group based on the one or more votes received; and providing, by the server, a highest ranked or scored Attribute Value for each attribute or the instance group in the portion of the plurality of attributes to the display interface for display. The method of claim 63, wherein the at least two of the one or more instances are grouped based on one or more of: respective Attribute Value, respective number of votes received, respective attribute category and respective rank or score of reliability.

A non-transient computer-readable medium containing program instructions, when executed by a processor, cause a computer to:

receive, from a plurality of data sources in real-time, and store in a data storage device, a plurality of data items and a plurality of attributes for each of the plurality of data items, each attribute having one or more instances, each instance having an Attribute Value; receive, from a display interface, a query to display a portion of the plurality of attributes; receive, from the display interface, one or more votes from one or more users for one or more Attribute Values for one or more of the plurality of data items;

determine a rank or score of reliability for each of the one or more Attribute Values based on the one or more votes received; and

provide a highest ranked or scored Attribute Value for each attribute in the portion of the plurality of attributes to the display interface for display.

The non-transient computer-readable medium of claim 65, wherein each Attribute Value has a corresponding data source of the plurality of data sources, and the computer is further caused to determine a rank or score of reliability for each of the plurality of data sources, and determine the rank or score of reliability for each of the one or more Attribute Values based on the rank or score of reliability for the corresponding data source.