Commit Graph

148 Commits

Author SHA1 Message Date
alexander.akait 75e3ea5c84 refactor: more types for serialization 2023-05-22 21:32:23 +03:00
alexander.akait 635d102e79 fix: types 2023-04-22 02:30:02 +03:00
Tobias Koppers 4a53e9af65 limit writes to
* chunks of 511MiB and
* total of 2GiB
2022-02-15 10:28:18 +01:00
Ivan Kopeykin 339af7e4f5 fix too big cache pack writing
if cache pack is too big, we should write in stream using callback
2022-02-11 16:16:48 +03:00
Ivan Kopeykin dfbce79f3c fix: provide correct serialization context
instead of context use local variable ctx
2021-10-11 13:41:31 +03:00
Tobias Koppers 629ac95660 write short strings more efficient to cache 2021-09-24 14:33:10 +02:00
Tobias Koppers 4a8bf4c084 allow to configure all hash functions used
fix default hash function for HttpUriPlugin to sha512
2021-09-23 14:55:13 +02:00
Tobias Koppers 3270275ffe split fresh packs by time
add comments

fix loosing cache

correctly store a lazy
2021-09-08 16:27:55 +02:00
Tobias Koppers db9b2df6cd improve allocations needed in serialization 2021-08-10 18:41:02 +02:00
Tobias Koppers b6609a4f6b add `experiments.buildHttp`
allows to build remote resources
stores remote resources in a lockfile
2021-08-05 15:47:18 +02:00
Tobias Koppers 492055ae90 add cache compression 2021-07-02 07:05:13 +02:00
Tobias Koppers 563a28dd0f fix toString for large positions 2021-06-29 13:37:34 +02:00
Tobias Koppers 0433e0b0e9 join sections 2021-06-29 13:09:01 +02:00
Tobias Koppers 86203ade47 use Buffer.from instead of slice
as slice doesn't work correctly in some environments for unknown reasons
2021-06-29 13:09:01 +02:00
Tobias Koppers 894ef94b86 use Buffer.from instead of slice
as slice doesn't work correctly in some environments for unknown reasons
2021-06-29 10:55:52 +02:00
Tobias Koppers e4389a494b
Merge pull request #13467 from myNameIsDu/fix/check-promise-condition
fix:the judgment of promise is changed from instanceof to Promises/A+…
2021-06-18 12:26:02 +02:00
Tobias Koppers e348fd61b6 reduce algorithmic complexity of the structure analysis for plain objects serialization 2021-06-18 10:44:35 +02:00
Tobias Koppers 42d882c2c2 shorter code 2021-06-18 10:21:19 +02:00
Tobias Koppers 3b555175e6 unreference some of the intermediate data when unserializing 2021-06-17 11:25:49 +02:00
du 946e96f476 fix:the judgment of promise is changed from instanceof to Promises/A+ standard 2021-05-29 00:53:19 +08:00
Tobias Koppers 4845915c0b Lazy require some modules 2021-05-26 21:35:46 +02:00
Tobias Koppers b0fa9ccada pretty files 2021-05-11 09:31:46 +02:00
Tobias Koppers 177736f59c avoid leaking unused memory in buffer backing stores in development mode 2021-04-20 15:22:16 +02:00
Tobias Koppers bf3f89c6ab create functions outside of context to avoid leaking the context 2021-04-20 15:17:31 +02:00
Tobias Koppers 43f9320e83 remove large arrays from context to avoid leaking 2021-04-20 14:13:04 +02:00
Tobias Koppers ff6656338d fix performance regression 2021-04-18 20:28:39 +02:00
Tobias Koppers 9cecf3c4a4 add support for serializing circular values 2021-04-14 08:56:32 +02:00
Tobias Koppers 51bd62d0b8 remove unused method 2021-04-08 18:26:22 +02:00
Tobias Koppers c1b8a6c859 reduce number of write syscalls by creating larger buffers
and may copy small existing buffers into the larger one
2021-04-08 18:26:22 +02:00
Tobias Koppers 2696446edf avoid memory leak in v8 ICs after (de)serializing 2021-03-25 13:24:52 +01:00
Tobias Koppers 3f378d9ff4 improve cache serialization by 30% 2021-03-22 20:28:08 +01:00
Tobias Koppers 9a7ffbabd3 only execute loader/require when no serializer is registered yet 2021-02-14 19:50:10 +01:00
Tobias Koppers 92fba2d346 fix json related tree shaking and caching problems
json with constructor property now caches correctly
json generation with object prototype property now works correctly
tree shaking json array when accessing length property now works correctly
fix incorrect call context when calling properties on imports from CommonJS modules

fixes #12615
2021-02-07 16:43:44 +01:00
Tobias Koppers 29d957d1cd fix allocation sizes 2021-02-03 11:53:08 +01:00
Tobias Koppers a1515fa9f0 add experimental lazy compilation 2021-01-21 17:39:56 +01:00
Tobias Koppers a2e326129c make filesystem types compatible with memfs and normal fs
fixes #12400
2021-01-13 10:14:58 +01:00
highonweb bc1911bf78 Changed name of memorize function to memoize 2020-12-27 03:02:57 +05:30
Tobias Koppers e500ad0714 nitpicks 2020-12-14 23:05:40 +01:00
Tobias Koppers 3db039a183 improve serialization of lazy elements 2020-12-14 16:43:29 +01:00
Tobias Koppers 9563338d90 improve reading of cache file 2020-12-14 13:16:20 +01:00
Tobias Koppers 005bdbbefd fix memory leak when using createReadStream 2020-12-14 11:06:15 +01:00
Tobias Koppers 1b00c9986f add polyfills for reading 64bit numbers from buffer for node 10 2020-12-14 10:48:21 +01:00
Tobias Koppers 6735b57888 use a dispatch table for performance 2020-12-14 08:16:10 +01:00
Tobias Koppers 8db0edbbcc microoptimization 2020-12-14 08:16:10 +01:00
Tobias Koppers f3bd8eda5b avoid references for single char strings 2020-12-14 08:16:10 +01:00
Tobias Koppers c194abeb65 dedupe Buffers during serialization 2020-12-14 08:16:09 +01:00
Tobias Koppers 227c16b9a8 improve string (de)serialization 2020-12-14 08:16:09 +01:00
Tobias Koppers 57c3a7eb05 optimize headers and encoding 2020-12-14 08:16:09 +01:00
Tobias Koppers 5a80ba12f1 help the GC 2020-12-14 08:16:09 +01:00
Tobias Koppers 4d1e913f48 avoid copy of data 2020-12-14 08:16:09 +01:00
Tobias Koppers c6311bae62 allow bigger cache sizes 2020-12-14 08:16:08 +01:00
Tobias Koppers 35363408c6 fix newly found eslint problems 2020-11-26 11:13:17 +01:00
Tobias Koppers 369dfe8997 reduce number of small buffers in BinaryMiddleware 2020-08-24 18:16:56 +02:00
Tobias Koppers 0701f48294 improve error message for weird objects in ObjectMiddleware
add Date serializer to default serializers
2020-06-02 16:09:48 +02:00
Tobias Koppers 49a6184e20 add tests to increase code coverage 2020-04-16 22:20:04 +02:00
evilebottnawi dafa4cd69a chore: setup cspell 2020-03-12 19:51:26 +03:00
Tobias Koppers ba832ec08c
Merge pull request #10326 from vankop/refactor-binary-middleware
refactor BinaryMiddleware
2020-02-06 09:42:15 +01:00
Ivan Kopeykin 19f5cd879e (#10326) resolve discussions 2020-02-05 13:19:10 +03:00
Ivan Kopeykin 350e397f63 refactor(FileMiddleware): add format information, minor code refactoring 2020-01-31 19:53:50 +03:00
Ivan Kopeykin 3cd865d7ab refactor(BinaryMiddleware): 2020-01-31 16:52:57 +03:00
Tobias Koppers eab25de485 dedupe object keys of plain object 2020-01-28 11:35:18 +01:00
Tobias Koppers f49838c95d style improvements and test 2020-01-28 11:35:10 +01:00
Tobias Koppers b40a345451 refactor Persistent Cache
cache is split into multiple .pack files which represent usage of cache items
this allows to only read/deserialize files that are needed
and reduces the need to rewrite .pack files
while still packaging multiple cache items together

lazy (de)serialization system has changed to allow writeLazy and writeSeparate
2020-01-28 11:35:06 +01:00
Tobias Koppers 95297186d5 remove Array.prototype.reduce calls to simplify code 2020-01-20 14:41:39 +01:00
Tobias Koppers b086bfbcc6 faster object type lookup during deserialization 2019-11-13 12:18:04 +01:00
Tobias Koppers 6eb9958643 print warning for large serialized strings 2019-11-10 21:04:54 +01:00
Tobias Koppers 8c594a61c1 Avoid recursion for nested plain objects 2019-11-10 21:04:49 +01:00
Tobias Koppers d8c93b40be faster serialization for deserialized data 2019-11-04 10:46:33 +01:00
Tobias Koppers 83c5305872 improve error message when deserialization failed 2019-11-01 12:19:27 +01:00
Tobias Koppers e9596ac799 faster referenceables in ObjectMiddleware 2019-08-13 17:03:01 +02:00
Tobias Koppers 563f8635b6 performance and memory improvement of BinaryMiddleware 2019-08-13 17:02:57 +02:00
Tobias Koppers 5320fcfbbe bugfix for nested promises in FileMiddleware 2019-08-13 17:02:47 +02:00
Tobias Koppers b2e7837288 replace Object.assign with object spread 2019-06-19 13:16:05 +02:00
Tobias Koppers 6db94536ad Merge tag 'v4.34.0' into next
4.34.0
2019-06-13 10:51:12 +02:00
Tobias Koppers e9c0d068dd refactor filesystem interfaces
add Compiler.intermediateFileSystem
avoid using `path` and `fs` module when possible
move `join`, `mkdirp` and `dirname` into utils
join and dirname is optional in FileSystem interface
remove mkdirp from Filesystem interface
2019-06-11 17:32:58 +02:00
Tobias Koppers e9c6539645 change Serializer API to be not file serialization specific 2019-01-24 17:42:38 +01:00
Tobias Koppers ad6e095c43 Improve typings of SerializerMiddlewares 2019-01-24 16:51:05 +01:00
Tobias Koppers 3a1eefec0a Measure size of objects to decide about inline or lazy storing 2019-01-24 15:14:41 +01:00
Tobias Koppers aa72c93b4d Use Buffer.allocUnsafe for better performance and buffer pooling 2019-01-24 15:14:39 +01:00
Tobias Koppers f68d465afa make reading from file safer and handle unexpected end of file 2019-01-19 11:49:30 +01:00
Tobias Koppers 00065741d2 log a warning when serialization fails 2019-01-19 11:47:19 +01:00
Sergey Tatarintsev eeb6e2db17 Improve memory consumption of file middleware
Buffers.concat creates a new buffer. On a large project, when using resolver
cache and pack store, total size of buffer can reach hundreds of megabytes. Use
streaming write instead.
2019-01-14 19:20:28 +01:00
Tobias Koppers 44663f8c67 allow serializing of objects with null prototype 2018-12-25 22:08:35 +01:00
Tobias Koppers d9238f061d memorize lazy data on access
remove need for read on write
add queue for file accesses in FileMiddleware to support concurrent access
2018-12-21 12:05:29 +01:00
Tobias Koppers aedc34b76d load internal serializers internally
allows to bundle webpack
2018-12-20 11:04:31 +01:00
Florent Cailhol 0d7889faab Remove unneeded intermediate objects 2018-11-02 11:57:42 +01:00
Tobias Koppers b1073349eb reorder maps to make binary encoding more efficient 2018-10-30 09:54:47 +01:00
Tobias Koppers 590772b934 add packed store mode 2018-10-30 09:54:40 +01:00
Tobias Koppers c46d89b02d serialize JS Errors 2018-10-25 10:52:33 +02:00
Michael Ciniawsky f7df43d080 feat(ErrorObjectSerializer): make `{Error}` serializable 2018-10-25 10:52:32 +02:00
Tobias Koppers 4b69df1117 allow to make classes not-serializable
make ContextModule not-serializable because it's not needed
2018-10-18 15:52:22 +02:00
Tobias Koppers 06cd79f17f add serializer for RegExp 2018-10-18 12:44:38 +02:00
Tobias Koppers 7786a6a0ed fix serializing of Buffers 2018-10-17 22:16:15 +02:00
Tobias Koppers b93ba710df add error for circular references while serializing 2018-10-17 21:25:46 +02:00
Tobias Koppers 2cfbea35bc fix bug with too long short strings 2018-10-17 21:25:41 +02:00
Tobias Koppers a9554a70e6 skip reference for empty strings 2018-10-17 21:25:40 +02:00
Tobias Koppers e1adc52745 improve length of emitted cache files 2018-10-17 21:25:40 +02:00
Tobias Koppers a852828279 add persistent caching prototype 2018-10-17 21:25:39 +02:00