CN102831245A - Real-time data storage and reading method of relational database - Google Patents
Real-time data storage and reading method of relational database Download PDFInfo
- Publication number
- CN102831245A CN102831245A CN2012103444607A CN201210344460A CN102831245A CN 102831245 A CN102831245 A CN 102831245A CN 2012103444607 A CN2012103444607 A CN 2012103444607A CN 201210344460 A CN201210344460 A CN 201210344460A CN 102831245 A CN102831245 A CN 102831245A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- algorithm
- time data
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a real-time data storage and reading method of a relational database, and the method comprises the following steps of real-time data storage: receiving external data; searching a data cache block corresponding to the received data; utilizing the data recorded by the data cache block to judge whether the currently received data is required to record or not; writing the currently received data into the data cache block; judging whether the data cache block is filled or not; submitting the data in the data cache block to a database engine; and real-time data reading: transmitting a data inquiry request; searching a data cache block corresponding to the inquired data; judging whether the currently inquired data is completely filled in the data cache block; submitting an inquiry request to the database engine, and writing the data returned by the database engine into a temporary cache zone; restoring corresponding data; and merging the data in the data cache block and the data in the temporary cache zone, and returning the inquiry result. Due to the adoption of the method, the problems that the traditional relational database is slow in reading-writing speed and poor in access performance can be solved.
Description
Technical field
The present invention relates to a kind of data of database storage and read method, particularly a kind of real-time data memory of relevant database and read method.
Background technology
Enterprise production process is the dynamic process that constantly changes, and the procedure parameter of sign production run such as temperature, pressure, flow, liquid level or the like are constantly to change; The enterprise production process material base that exists of relying is a production equipment, and the operational factor of production equipment such as state, electric current, voltage, power, vibration index or the like are also constantly changing; Residing environmental baseline of production run such as atmospheric pressure, temperature, humidity, inflammable and explosive, poisonous, harmful gas concentration or the like are constantly changing especially.
The production run real time data is undoubtedly one of the most valuable information resources of enterprise.Any have the enterprise of enterprising spirit all to hope intactly storing process data; Thereby went the efficient and the quality situation of analytic process operation from " past " and " history " of production run; And therefrom find potential hidden danger, to improve the management level and the technology level of enterprise.The characteristic that process real-time information is huge, content is fast changing has been brought unprecedented challenge to data storage technology.
In the first phase monograph of the ACM SIGMOD Record that the notion of real-time data base RTDB (Real-Time Data Base) appears in March, 1988 the earliest.Indicate the establishment of this emerging research field of real-time data base.Real-time data base is a branch of Database Systems development, is the fusion of real-time domain and database field, and it is applicable to the issued transaction of handling the fast-changing data of bringing in constant renewal in and having time restriction.The real-time data base technology is the product that real-time system and database technology combine; The researchist hopes to utilize database technology to solve the data management problem in the real-time system, utilizes real-time technique to drive scheduling and resource allocation algorithm for real-time data base provides the time simultaneously.Yet real-time data base is not to be both simply integrated on notion, structure and method.
Early stage infosystem receives the restriction of computer hardware, software engineering level; Can't solve the storage problem of real time data with the relevant database storage engines that service-oriented is handled; Has only the man specialized real-time database of number manufacturer at present; The storage solution of real time data is provided, like the PI real-time data base of U.S. OSIsoft, the IP21 of Aspen Technology etc.Because be " specialty " product, do not pass through the abundant baptism in business software market, so its product price is very expensive, large enterprise implements real time information system, and only the cost input of real time data library software maybe be all more than 1,000,000 yuans.
According to the interrelated data statistics, a main barrier of present domestic enterprise production process informationization is the high acquisition cost and the total cost of ownership (TCO) of real-time data base software product just.
In recent years, under the propelling of IT application in enterprises tide, the relevant database engine technique has obtained widely using, its software product maturation that in the experience and tempering in market, becomes better and approaching perfection day by day.The IT personnel of enterprise are also handy day by day to application, the maintenance of relevant database software product.
Relevant database engine and real-time data base engine all are to be used for storing data, and to the user reliable, available data access function are provided with the highest performance, carry out the automatic management of storage system resource simultaneously automatically.But serious difference is arranged on application scenarios:
The storage size relevant database was mainly used in the business process data of storage enterprise originally, and business process is the process that needs the people to participate in, and large supermarket's per second the busiest needs data recorded also can not surpass the hundreds of bar.Otherwise; The real-time data base engine is mainly used in the production run real time data of storage enterprise; Development along with automated control technology; Mostly the production run of modern enterprise is the process in the continuous operation of opertaing device control serialization down, and the data that typical modern chemical industry device per second generation is up to ten thousand have been comparatively common situation.
Real-time is because business process needs the people to participate in; A business process needs several minutes time from start to end at least; Want time several seconds at least, and the modern production process is automated procedure basically, in the time of Millisecond, just can accomplishes a process.
Illustrate; If we will store the sampled data of 1 time/per second of 10000 work station points; Storing the disk storage scale that needed in 1 year is: the quality sign indicating number of data needs 1 byte; The timestamp record sampling time squints and is accurate to millisecond needs 3 bytes, and it is 4 bytes that data are assumed to single precision floating datum, and promptly writing down a single precision floating datum needs 8 bytes.
8X10000X3600X24X365≈2350GB≈2.29TB
Average every millisecond needs 10 data of record
The storage size of real time data engine and real-time all are very surprising.
From bottom technology realization aspect, all the use a computer file system management function of operating system of relevant database and real-time data base is come access data, and there are not the special technique means that directly arrive disk in real-time data base.Because the main flow large capacity disc still mechanically rotates at present, so file I/O is consuming time.Improve access performance, set about from the efficient buffering of file system, this all is consistent to relevant database and real-time data base.System R adopts transaction management to guarantee the integrality of data, and same real-time data base can not be ignored the integrality of data.Therefore, realization is tight, issued transaction is the common target of relevant database and real-time data base efficiently.From concurrent access control, the target of two kinds of system's pursuits all is consistent, promptly bigger safe concurrency certainly.Others; Like index structure, access mechanism, Restoration Mechanism, log management etc.; Because relevant database has more " market maturity ", the relevant database product is having better performance aspect availability, stability, scalability, the cost performance.
Think in the industry and can't use relational data engine storage of real time data, reason is:
The relevant database read or write speed is slow, and access performance is poor
Relevant database storage efficiency low (can not packed data)
---select from internet " the simple comparison of relational database, memory database, real-time data base "
The experiment of readwrite performance problem shows, uses relational data library storage magnanimity, successional real time data to tend to cause the serious deterioration of database engine readwrite performance in due form.Discover that the deterioration of database engine performance is caused by the storage architecture design of mistake.Suppose to use relevant database engine storage of real time data, can design following list structure:
This data structure that can adopt usually exists serious performance hidden danger; Insert (insert) for data; Modern relevant database engine can reach the performance of several ten thousand row/seconds fully; Load at the high capacity rapid data that uses database engine to provide under the situation of interface, data are inserted and are not had performance bottleneck, can satisfy the storage demand of large-scale process enterprises several ten thousand point～tens0000 real time data fully.But such table storage organization design can cause data query (reading) the performance performance of very severe.
Through the main applicable cases analysis to real-time data base, real time information system to the basic mode of the historical query of process data is: the numerical value of inquiring about one (or a plurality of item) all storages in certain hour section (a few minutes or several hours).
In disk file system, basic storage cell is the sector, at present; The sector-size of hard disk is generally 512 bytes; File management system must be the unit access data with the sector, if data in the different sector stored, then must moving head; Moving head is a mechanical process, so the file access operation is " machinery " consuming time operation.
Database engine is the unit management database file with the data page; The size of a data page is for being generally 8KB; With the above-mentioned data structure storage real time data of 10,000 point/seconds; Can cause the different data storage constantly of an item in different pieces of information page or leaf different disk sector, suppose that 1 hour every work station point has write down 3600 records (1 time/second sampling rate), the historical data that only reads 1 hour 1 work station point will cause minimum magnetic head tracking and magnetic head more than 3600 times to move; Thereby cause the serious deterioration of data query performance, as shown in Figure 1.
Storage efficiency problem storage efficiency is meant limited disk space storage more data.Storage efficiency is most important to the storage of real time data.Unlike the database application of service-oriented, the data of real time information system are continually, every minute and second do not stop " automatically " produces from the process apparatus of enterprise.If we will store the sampled data of 1 time/per second of 10000 work station points, store the disk space that needed in 1 year:
The quality sign indicating number of data needs 1 byte, and the timestamp record sampling time squints and is accurate to millisecond needs 3 bytes, and it is 4 bytes that data are assumed to single precision floating datum, and promptly writing down a single precision floating datum needs 8 bytes.
8X10000X3600X24X365≈2350GB≈2.29TB
The demand of memory capacity is quite surprising; If can improve storage efficiency; Not only can reduce the hardware acquisition cost of enterprise information system, the maintenance cost of system, the raising of storage density also can bring comprehensive lifting for the access performance of real-time data memory system.
Summary of the invention
Goal of the invention: to the problem and shortage of above-mentioned prior art existence; The real-time data memory and the read method that the purpose of this invention is to provide a kind of relevant database, the problem that read or write speed is slow, access performance is poor, storage efficiency is low when solving the extensive real time data of traditional relational database access.
Technical scheme: for realizing the foregoing invention purpose, the technical scheme that the present invention adopts is a kind of real-time data memory and read method of relevant database, comprising:
(1) real-time data memory:
Step 1: receive data from the outside;
Step 2: search the metadata cache piece corresponding with the data that receive;
Step 3: utilize metadata cache piece data recorded to use first algorithm to judge whether the data of current reception need record,, then return step 1,, continue step 4 if need record if do not need record;
Step 4: the data of current reception are write the metadata cache piece;
Step 5: whether the judgment data cache blocks has been write full, expires if write, and returns step 1, expires if write, and continues step 6;
Step 6: submit the data in the metadata cache piece to database engine;
(2) real time data reads:
Step 1): send the data query request;
Step 2): search the metadata cache piece corresponding with the data of inquiring about;
Step 3): whether the data of judging current inquiry if the data of inquiry all in the metadata cache piece, are then returned Query Result, are returned step 1), otherwise are continued step 4) all in the metadata cache piece;
Step 4): submit query requests to database engine, and the data that database engine returns are write interim buffer area;
Step 5): do not need data recorded if exist in the step 3, then adopt second algorithm to recover corresponding data;
Step 6): if in the metadata cache piece in the step 3) data are arranged, then the data in pooled data cache blocks and the interim buffer area are returned Query Result.
Further, in the said step 6, use the data in the algorithm packed data cache blocks earlier, submit the data in the metadata cache piece to database engine again; In the said step 4), after the data that database engine is returned write interim buffer area, use the algorithm data in the interim buffer area that decompress.
Further, in the said step 2, use Index Algorithm to search the metadata cache piece corresponding with the data that receive.
Further, said Index Algorithm is the RBTree Index Algorithm.
Further, said first algorithm is a sectional broken line fitting data filter algorithm, and said second algorithm is the linear fit algorithm.
Further, said algorithm is the LZ compression algorithm.
Beneficial effect: the present invention has realized utilizing modern relevant database engine to store the overall solution of enterprise's magnanimity production process data.Use the memory cache technology to avoid the performance bottleneck of continuous data inquiry, utilize data filter technology and lossless compressiong to improve the storage efficiency and the density of data storage of infosystem.The present invention has realized utilizing the relevant database product to store the magnanimity real time data fast and efficiently.
Description of drawings
Fig. 1 is irrational real-time data memory synoptic diagram;
Fig. 2 is the synoptic diagram of sectional broken line fitting algorithm;
Fig. 3 is the synoptic diagram of lossless compress test data (sinusoidal curve superpose positive and negative 5% noise);
Fig. 4 is the process flow diagram of data storage;
The process flow diagram that Fig. 5 reads for data.
Embodiment
Below in conjunction with accompanying drawing and specific embodiment; Further illustrate the present invention; Should understand these embodiment only be used to the present invention is described and be not used in the restriction scope of the present invention; After having read the present invention, those skilled in the art all fall within the application's accompanying claims institute restricted portion to the modification of the various equivalent form of values of the present invention.
The present invention has realized using the technical scheme of relevant database engine storage enterprise magnanimity real time data, and gordian technique is:
Adopt sectional broken line fitting data filter algorithm, raw data is filtered, can pass through the redundant data of algorithm reduction, thereby improve the storage efficiency of data with filtering.
In data processing field, there is the several data filter algorithm at present, like change threshold, linear fit, conic fitting or the like.Characteristic and algorithm taking all factors into consideration according to the production process data variation to the computing power demand; Adopted " sectional broken line match " algorithm; Its algorithm core is with straight line a series of process datas of match in the error tolerance band; And straight line only need write down 2 points, thereby keeps important delta data, filters out the redundant data that repeat or can be rebuild by algorithm.As shown in Figure 2, first sampled point of supposing the system is A, because be initial value; Therefore the A point is marked as and needs record, should deposit database in, when receiving the 3rd sampled point C; Draw a straight line from the A point to the C point, calculate the error e between the B time point on B point actual value and the straight line, if e is in the tolerance band of default; The B point just is marked as and need not record, and same C point also is marked as and need not record.When receiving the E point data, according to the broken line slope data of storing in the computation process, judge D point actual value and A and surpassed e, so the D point needs record to the error of E point fitting a straight line point, D point carries out next one judgement and circulates as new initial point simultaneously.
Diagram shows that the point of data-base recording has only A, D, E, F, G, the numerous points in the real process curve; Like B, C point all by filtering; When data read the reduction reconstruction, the segmented fitting real process curve of using A, D, E, F, G to order, error can not surpass the permissible error e value that can set.
The storage framework that utilizes the technology of metadata cache efficiently to change common relevant database engine is to make real time data reduces hard disk as far as possible when inquiry is read tracking and move; Guarantee query performance; The data of the different sampling stages of an item must leave in the continuous disk sector, and this is that " machinery " characteristic by disk file system determines.
The present invention has adopted the method for memory buffer to realize the continuous storage problem of real time data.At first; For each work station point distributes the core buffer (also claim the metadata cache piece, size is like 4KB or 8KB) of fixed size, the real time data that sampling obtains is not the disk that writes direct; But first write memory buffer zone (being called for short " buffer zone "); After buffer zone is write completely, write the content of whole buffer zone again to disk, like this each work station point store data in inherent disk sector continuously of a time period (how many data are the size that depends on buffer area can hold).
Such data page can be stored 30 minutes to 1 hour real time data of a work station point thick and fast, inquires about 1 hour data of a work station point and only need read 1～2 data page, and hard disc magnetic head moves with tracking and only need can accomplish for 1～2 time.Under this and the traditional scheme, inquire about 1 hour data of a work station point and need read at least 3600 data pages, carrying out minimum tracking process more than 3600 times has huge performance difference.
Also can realize above-mentioned storage organization at the relevant database engine.Modern data storehouse engine is generally supported big scale-of-two (BLOB) data type, with the continuous real time data of binary block storage, the list structure below using.
Notice that data are not stored in the index page; But be placed on LOB_DATA (large object data field), and only deposit the file pointer that points to data page at index page, reduced index page (B tree so greatly; Be binary tree) the sizes of memory demand, thereby better query performance is provided.
Use the data compression algorithm of efficient lossless that buffer data is compressed, further improve the density of data storage
One skilled in the art will appreciate that data compression is divided into two types of lossy compression method and lossless compress, as far as process data, lossy compression method can realize through following technology:
Reduce the precision of timestamp.Some manufacturer thinks, as far as process, it is in all senses useless to be accurate to millisecond, and therefore, timestamp is accurate to 100 milliseconds or 10 milliseconds, the storage space that can big time saver stabs.
Reduce the precision of data.Can use the floating number (8 byte) of the floating number replacement double precision of single precision (4 byte), represent normalized 4 byte single precision floating datums with 2 byte fixed-point numbers.
Do not store the quality sign indicating number, with the quality situation of the numeric representation data of data self.
The applicant thinks that the data compression that reduces data precision is worth choosing.If historical data just is used for the picture change trend curve, limited because of the screen resolution of computing machine, these ways that reduce data precisions are given no cause for much criticism.Look forward to the future, the purposes of process data most worthy possibly be that like association analysis, trend analysis, statistical study or the like, at this moment more high-precision data can have higher information value as data analysis.Moreover; An important development trend of modern process industry is exactly " becoming more meticulous "; 0.01 ℃ process data change and all might bring influence product quality, like the crystallization process of pharmaceutical industry, the modern process instrument also develops to digitizing, high precision direction on the other hand; The price of memory device is also more and more lower, is not the way that faces the future so reduce the data compression of data precision.
Practice shows that some current in the industry at present lossless data compression algorithms are the compression applications that can be applied to the process real-time data fully.What these algorithms were directed against all is the off-line compression; Through analyzing the file data blocks of specific size; Use " dictionary " thus find that with some other algorithm the rule of repetition or the repetition of data finds that the minimum redundancy of compressed data encodes; Therefore data block is big more, and it is big more redundant possibility to occur, and the redundance of data itself high (comprising a large amount of repeating datas) also means higher compression factor.
Real time data elder generation buffer memory is stored again; For utilizing lossless compression algorithm that real time data is carried out the possibility that lossless compress provides realization; Because discrete separately data do not have any redundance and can say; And data just possibly found the Changing Pattern that contains wherein behind memory cache, and lossless compression algorithm could utilize these rules to carry out data compression.
Here the applicant adopts improved Lempel-Ziv (LZ) compression method; Though carried out the preliminary filtration of raw data through " sectional broken line match "; The redundance of data reduces greatly, but practice shows that the LZ compression algorithm still can reach the compressibility about 50%.As shown in Figure 3, data are done sinusoidal fluctuation, and stack is with the small size noise of high frequency.The data buffer size is 8KB, and the data filter error is 0.5%, and the compressibility of LZ compression algorithm is 45%.
As shown in Figure 4; Data Receiving can be accepted the process data that the different data acquisition module collects; For improving concurrent performance, data transfer has adopted " postal delivery " mechanism in the system design, and promptly the data acquisition thread does not directly call follow-up data processing, memory function; But in the memory block " envelope " with the distribution of real time data writing system; Deliver and give special data transmission line journey, by sending follow-up data processing, the stored programme of thread dispatching, the data acquisition thread can be absorbed in the very strong data acquisition task of real-time again.
Because the work station point of need gathering is up to several ten thousand points～hundreds of thousands point, so need adopt efficiently the internal storage data Index Algorithm to locate the corresponding metadata cache piece of work station point fast.In the present embodiment, adopt the RBTree Index Algorithm to search.
Whether sectional broken line fitting data filter algorithm utilizes metadata cache piece data recorded to judge current data needs record, need not the algorithm interpolation reduction that data recorded can be used linear fit, need not to write disk; Need data recorded to write buffer area.
After buffer area is write full (8K size); Use LZ lossless compression algorithm (being called for short " LZ algorithm ") to compress to data cached; Submit the buffer area data after compressing to database engine then; To the new metadata cache piece of system's application (also claim data cached memory block, or be called for short buffer memory, buffer area), begin new storage circulation simultaneously.
As shown in Figure 5, the data query request can come from the data, services interface that system provides.
Because the caching technology that adopts; Therefore nearest real time data might not submitted to database as yet; Data query should at first be searched data through internal storage data Index Algorithm (present embodiment is the RBTree Index Algorithm) in buffer area; If the data of inquiry all in buffer area, can directly be returned Query Result, needn't carry out disk I operation consuming time again.
Submit the inquiry application to database engine, and the data that database engine returns are write interim buffer area.Because data-base recording is the data of recompile after the LZ compression algorithm, therefore need to use the data in the interim buffer area of LZ decompress(ion).
According to query argument, use the linear fit algorithm, simulate the data between the adjacent data RP, if also have partial data to be arranged in previous buffer area, also need merge these data, thereby return Query Result.
Practical application of the present invention is following:
SQL Server enterprise version database engine is adopted in the storage of real time data, and the real time data access module uses the .NET4.0C++/CLI establishment.
The storage specification
Running environment: processor:
Dual-Core E54002.7GHz, operating system: Windows Server 2003, Enterprise Edition2; Internal memory: 4GB; Hard disk: 500GB, database: SQL Server2005 enterprise version, network: 100M LAN; Real time data point: 23100; Sampling rate 1 time/second, the concurrent users number: maximum 100, average 65.
Following in above running environment reaches following performance index:
< 1000 milliseconds of real time data query responding times;
Data compression ratio 20:1～50:1;
The average operating load of CPU is not more than 20%.
Claims (6)
1. the real-time data memory of a relevant database and read method comprise:
(1) real-time data memory:
Step 1: receive data from the outside;
Step 2: search the metadata cache piece corresponding with the data that receive;
Step 3: utilize metadata cache piece data recorded to use first algorithm to judge whether the data of current reception need record,, then return step 1,, continue step 4 if need record if do not need record;
Step 4: the data of current reception are write the metadata cache piece;
Step 5: whether the judgment data cache blocks has been write full, expires if write, and returns step 1, expires if write, and continues step 6;
Step 6: submit the data in the metadata cache piece to database engine;
(2) real time data reads:
Step 1): send the data query request;
Step 2): search the metadata cache piece corresponding with the data of inquiring about;
Step 3): whether the data of judging current inquiry if the data of inquiry all in the metadata cache piece, are then returned Query Result, are returned step 1), otherwise are continued step 4) all in the metadata cache piece;
Step 4): submit query requests to database engine, and the data that database engine returns are write interim buffer area;
Step 5): do not need data recorded if exist in the step 3, then adopt second algorithm to recover corresponding data;
Step 6): if in the metadata cache piece in the step 3) data are arranged, then the data in pooled data cache blocks and the interim buffer area are returned Query Result.
2. according to the real-time data memory and the read method of the said a kind of relevant database of claim 1, it is characterized in that: in the said step 6, use the data in the algorithm packed data cache blocks earlier, submit the data in the metadata cache piece to database engine again; In the said step 4), after the data that database engine is returned write interim buffer area, use the algorithm data in the interim buffer area that decompress.
3. according to the real-time data memory and the read method of the said a kind of relevant database of claim 1, it is characterized in that: in the said step 2, use Index Algorithm to search the metadata cache piece corresponding with the data that receive.
4. according to the real-time data memory and the read method of the said a kind of relevant database of claim 3, it is characterized in that: said Index Algorithm is the RBTree Index Algorithm.
5. according to the real-time data memory and the read method of the said a kind of relevant database of claim 1, it is characterized in that: said first algorithm is a sectional broken line fitting data filter algorithm, and said second algorithm is the linear fit algorithm.
6. according to the real-time data memory and the read method of the said a kind of relevant database of claim 2, it is characterized in that: said algorithm is the LZ compression algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210344460.7A CN102831245B (en) | 2012-09-17 | 2012-09-17 | A kind of real-time data memory of relevant database and read method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210344460.7A CN102831245B (en) | 2012-09-17 | 2012-09-17 | A kind of real-time data memory of relevant database and read method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102831245A true CN102831245A (en) | 2012-12-19 |
| CN102831245B CN102831245B (en) | 2015-08-05 |
Family
ID=47334380
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210344460.7A Expired - Fee Related CN102831245B (en) | 2012-09-17 | 2012-09-17 | A kind of real-time data memory of relevant database and read method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102831245B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103186673A (en) * | 2013-03-30 | 2013-07-03 | 马钢控制技术有限责任公司 | Massive data storage and inquiry method |
| CN105138543A (en) * | 2015-07-09 | 2015-12-09 | 广州杰赛科技股份有限公司 | Data storage method and system |
| CN105589884A (en) * | 2014-10-24 | 2016-05-18 | 腾讯科技（深圳）有限公司 | Data processing method and data processing device |
| CN105631035A (en) * | 2016-01-04 | 2016-06-01 | 北京百度网讯科技有限公司 | Data storage method and device |
| CN105740293A (en) * | 2014-12-12 | 2016-07-06 | 金蝶软件（中国）有限公司 | Data export method and device |
| CN106339374A (en) * | 2015-07-06 | 2017-01-18 | 天津九洲云物联科技有限公司 | Embedded system file type real time history database algorithm |
| CN106649828A (en) * | 2016-12-29 | 2017-05-10 | 中国银联股份有限公司 | Data query method and system |
| CN106815227A (en) * | 2015-11-27 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of storage information |
| CN108073706A (en) * | 2017-12-20 | 2018-05-25 | 北京四方继保自动化股份有限公司 | A kind of method of analogue system history library longitudinal data transverse directionization displaying |
| CN109033347A (en) * | 2018-07-23 | 2018-12-18 | 广州大学 | A kind of real-time quick collision method of multidate information based on red black tree |
| CN110209640A (en) * | 2019-06-06 | 2019-09-06 | 四川长虹电器股份有限公司 | The method of switching at runtime lz4 compression algorithm type under cell phone system operating status |
| CN110309137A (en) * | 2018-02-28 | 2019-10-08 | 贵州白山云科技股份有限公司 | A kind of data managing method and device |
| CN110647588A (en) * | 2018-06-08 | 2020-01-03 | 贵州高新翼云科技有限公司 | Universal algorithm for automatically filling all forms in any complex relation data model and automatically testing big data |
| CN111324455A (en) * | 2020-02-10 | 2020-06-23 | 浙江中智达科技有限公司 | Industrial cloud platform scheduling method, device and system |
| CN113687773A (en) * | 2021-07-16 | 2021-11-23 | 深圳智慧林网络科技有限公司 | Data compression model training method and device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020046204A1 (en) * | 2000-08-25 | 2002-04-18 | Hayes Scott R. | Heuristic automated method for ideal bufferpool tuning in a computer database |
| CN101604311A (en) * | 2008-11-25 | 2009-12-16 | 无锡虹业自动化工程有限公司 | Breaking away industrial configuration software database interface is realized the method for historical data report |
| CN101667205A (en) * | 2009-09-28 | 2010-03-10 | 河南电力试验研究院 | Method for memorizing real time measure point data facing quick review |
| CN102611454A (en) * | 2012-01-29 | 2012-07-25 | 上海锅炉厂有限公司 | Dynamic lossless compressing method for real-time historical data |
-
2012
- 2012-09-17 CN CN201210344460.7A patent/CN102831245B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020046204A1 (en) * | 2000-08-25 | 2002-04-18 | Hayes Scott R. | Heuristic automated method for ideal bufferpool tuning in a computer database |
| CN101604311A (en) * | 2008-11-25 | 2009-12-16 | 无锡虹业自动化工程有限公司 | Breaking away industrial configuration software database interface is realized the method for historical data report |
| CN101667205A (en) * | 2009-09-28 | 2010-03-10 | 河南电力试验研究院 | Method for memorizing real time measure point data facing quick review |
| CN102611454A (en) * | 2012-01-29 | 2012-07-25 | 上海锅炉厂有限公司 | Dynamic lossless compressing method for real-time historical data |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103186673A (en) * | 2013-03-30 | 2013-07-03 | 马钢控制技术有限责任公司 | Massive data storage and inquiry method |
| CN105589884A (en) * | 2014-10-24 | 2016-05-18 | 腾讯科技（深圳）有限公司 | Data processing method and data processing device |
| CN105589884B (en) * | 2014-10-24 | 2020-11-03 | 腾讯科技（深圳）有限公司 | Data processing method and device |
| CN105740293B (en) * | 2014-12-12 | 2019-07-23 | 金蝶软件（中国）有限公司 | Data export method and device |
| CN105740293A (en) * | 2014-12-12 | 2016-07-06 | 金蝶软件（中国）有限公司 | Data export method and device |
| CN106339374A (en) * | 2015-07-06 | 2017-01-18 | 天津九洲云物联科技有限公司 | Embedded system file type real time history database algorithm |
| CN105138543A (en) * | 2015-07-09 | 2015-12-09 | 广州杰赛科技股份有限公司 | Data storage method and system |
| CN106815227A (en) * | 2015-11-27 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of storage information |
| CN105631035A (en) * | 2016-01-04 | 2016-06-01 | 北京百度网讯科技有限公司 | Data storage method and device |
| CN105631035B (en) * | 2016-01-04 | 2018-04-17 | 北京百度网讯科技有限公司 | Date storage method and device |
| CN106649828A (en) * | 2016-12-29 | 2017-05-10 | 中国银联股份有限公司 | Data query method and system |
| CN106649828B (en) * | 2016-12-29 | 2019-12-24 | 中国银联股份有限公司 | Data query method and system |
| CN108073706B (en) * | 2017-12-20 | 2022-02-22 | 北京四方继保自动化股份有限公司 | Method for transversely displaying longitudinal data of simulation system historical library |
| CN108073706A (en) * | 2017-12-20 | 2018-05-25 | 北京四方继保自动化股份有限公司 | A kind of method of analogue system history library longitudinal data transverse directionization displaying |
| CN110309137A (en) * | 2018-02-28 | 2019-10-08 | 贵州白山云科技股份有限公司 | A kind of data managing method and device |
| CN110647588A (en) * | 2018-06-08 | 2020-01-03 | 贵州高新翼云科技有限公司 | Universal algorithm for automatically filling all forms in any complex relation data model and automatically testing big data |
| CN109033347A (en) * | 2018-07-23 | 2018-12-18 | 广州大学 | A kind of real-time quick collision method of multidate information based on red black tree |
| CN110209640A (en) * | 2019-06-06 | 2019-09-06 | 四川长虹电器股份有限公司 | The method of switching at runtime lz4 compression algorithm type under cell phone system operating status |
| CN111324455A (en) * | 2020-02-10 | 2020-06-23 | 浙江中智达科技有限公司 | Industrial cloud platform scheduling method, device and system |
| CN113687773A (en) * | 2021-07-16 | 2021-11-23 | 深圳智慧林网络科技有限公司 | Data compression model training method and device and storage medium |
| CN113687773B (en) * | 2021-07-16 | 2023-08-11 | 深圳智慧林网络科技有限公司 | Data compression model training method and device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102831245B (en) | 2015-08-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102831245B (en) | A kind of real-time data memory of relevant database and read method | |
| CN109684352B (en) | Data analysis system, data analysis method, storage medium, and electronic device | |
| US11132384B2 (en) | Generating a multi-column index for relational databases by interleaving data bits for selectivity | |
| Jensen et al. | Time series management systems: A survey | |
| Tan et al. | Clost: a hadoop-based storage system for big spatio-temporal data analytics | |
| Deri et al. | tsdb: A compressed database for time series | |
| CN103914485B (en) | System and method for remotely collecting, retrieving and displaying application system logs | |
| EP2572289B1 (en) | Data storage and processing service | |
| EP2040180B1 (en) | ETL-less zero-redundancy system and method for reporting OLTP data | |
| US10649991B2 (en) | Pruning of columns in synopsis tables | |
| EP2270692A1 (en) | Lifecycle-based horizontal partitioning | |
| CN102375853A (en) | Distributed database system, method for building index therein and query method | |
| CN110765154A (en) | Method and device for processing mass real-time generated data of thermal power plant | |
| CN102073712B (en) | Holographic process data archiving and inverting method based on dynamic change frame | |
| US20140258315A9 (en) | Method And Process For Enabling Distributing Cache Data Sources For Query Processing And Distributed Disk Caching Of Large Data And Analysis Requests | |
| CN102779138A (en) | Hard disk access method of real time data | |
| US11797506B2 (en) | Database management systems for managing data with data confidence | |
| Lu et al. | TridentKV: A read-Optimized LSM-tree based KV store via adaptive indexing and space-efficient partitioning | |
| CN107894875B (en) | RTU data storage method | |
| CN112650739A (en) | Data storage processing method and device for coal mine data middling station | |
| Li et al. | FluteDB: An efficient and scalable in-memory time series database for sensor-cloud | |
| CN111046054A (en) | Method and system for analyzing power marketing business data | |
| Jiadi et al. | Research on Data Center Operation and Maintenance Management Based on Big Data | |
| Murugesan et al. | Evaluate database compression performance and parallel backup | |
| Li et al. | A Comparative Study of Row and Column Storage for Time Series Data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20150805 Termination date: 20170917 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |