Use Onnx Model #9
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* fix bug where actions are duplicated when tracing torch functions on Placeholders * remove unused file
* Fix hook by removing unused and confusing functions * Fix hook of PlaceHolder and add tracer to th.tensor * Update list of ambiguous and hook-excluded functions * Improve instantiation of placeholder in Operation response * Fix extract_key bug, add checks and explicative failures * Add tests for plans with comparison and funcs with no args * Add experimental notebooks for back up * Add first version of FSS * Add dtype serialization and small fix in ptr plan * Add checks to AdditiveSharedTensor * Add clarification error to Plan * Add support of multi pointer tensors on plan ptr with>1 locations * Import placeholder instantiation * Update FSS * Run black * Improve FSS * Add a tag index for worker and a find_by_tag method * Clean object storge rm_obj and use tag index * Fix search behavior, optimize search using tag index * Update plans & pointerplans to support tag serialization * Clean & optimize FSS equality * Add DIF plan version * Add comparison on AST using FSS DIF * Reduce code duplication in FSS * Rm notebooks * Add the 'protocol' keyword to .share * Add support for all bool ops using FSS + add all tests * Fix tests on plans * Fix Protocol to pass tests * Fix baseworker search to handle empty response and empty query * Reduce calls to PRF * Basic FL training plan example * tmp work * Improve plan representation * Fix problems after merge * Tmp work * Add a crypto primitive store * Add option to get the result back of a remote command execution * Add an hybrid version of fake Plans using @tracer * Add fast version with mock plans * Add faster PRG and increase security parameter to 110 * Add comparison support and start xor to add sharing conversion * Add support of fast FFS eq & comp in 1D tensors * Merge conflicts, adapt to a more strict GC process, update tests * Update DPF to support >=1D tensors and add tests for DPF * Add test for EmptyCryptoException + transform crypto_store to a "butter like" store: there is a big tensor and you take part of it depending of the nb of prilmitives that you need * Add support for tensor in DIF class + tests * Add AST comparison using FSS on tensors * rename klass -> class_ * Implement move_ inplace * Update tutorial 9 on worker._objects * Fix tests * Add small caching mecanism * Add comments and clean code * Correct back remote_send * Tiny fix on primitive generation * Address review * Fix th.narrow and comments * Run black * Fix small bugs * Fix bugs Co-authored-by: Vova Manannikov <vmanannikov@broadsoft.com>
…enMined#3392) * Private method added * Field and torch_dtype for custom tensors * Description for custom dtype * Fix deserialization * Increase sigmoid exp tolerance (OpenMined#3427) * Fix double tracing bug (OpenMined#3428) * fix bug where actions are duplicated when tracing torch functions on Placeholders * remove unused file * Assert `dtype!="custom"` * Assertion msg * Assertion msg * Remove assertion for "core" operations * Apply suggestions from code review Single line comparision Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com> * Flexibility over int dtypes * Derp in field -_- * dtype fixed * Drop short and int8 support :) Serde ain't easy peasy Co-authored-by: George-Cristian Muraru <murarugeorgec@gmail.com> Co-authored-by: Jasopaum <jason.paumier@gmail.com> Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com>
* [WIP] Fix `Plan` tracing that involves `backward()`/autograd * Set `requires_grad` to `True` on `Plan` building temporary args * Initial attempt to autograd Plan tracing * Address review comments * Fix unit tests * Fix gc unit tests * Empty commit Co-authored-by: Vova Manannikov <vova@promail.spb.ru>
* Modified securenn to work with >3 parties * Improved comments and wrap * Cleaned some leftover code * Fixed a bug in maxpool_deriv * Added tests for securenn with >3 parties * Improved usability of maxpool_deriv * run black * Ran black on changed files * Fixed bug * ran black * fixed bug on tests * ran black, again * Move tests from torch to mpc Co-authored-by: Daniel Hurtado Ramírez <trdhrz@gmv.es> Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com>
* Replace `Protocol` code with `Plan` code (as a starting point) * Remove translation from `Protocol` for now * Set up `func2protocol` on `syft` module * Add a failing `Protocol` test * convert role attribute to a dict of Roles * handle input and output repartition * remove Protocol serde tests * remove pointer_protocol * remove protocol tutorial * remove references to PointerProtocol * remove references to PointerProtocol * fix and add test for Protocol.copy() * comment code still not used * update protocol serde for msgpack * update protocol protobuf serde * Plan and Protocol need to be built before serde * remove input_repartition and output_repartition * remove input/output_repartition from serde tests * remove is_built from serde tests * address review comments * remove last occurrences of state in protocols * remove include_state from tests * update syft-proto dep * remove make_ptr_protocol in serde tests Co-authored-by: Jasopaum <jason.paumier@gmail.com>
* Added autoscale , automatically spin-up instance * Added instructions to set up GCP account * reformatted * fixed the destroy method * moved to grid package
* Allowed users as tuple * Remove pdb
* cleanup rnn.py * clean up and comment * fix batch_first
* add Bengali translated notebook #Part 5. * add translator name. * add Bengali translated notebook #part - 4. * modify bengali translated Part 01 - The Basic Tools of Private Deep Learning notebook. * modify bengali translated Part 02 - Intro to Federated Learning notebook. * modify bengali translated 'Part 06 - Federated Learning on MNIST using a CNN' notebook * modify bengali translated 'Part 06 - Federated Learning on MNIST using a CNN' notebook * modify bengali translated notebook 'Part 07 - Federated Learning with Federated Dataset'. * modify bengali translated notebook Part 08 - Introduction to Plans. * modify bengali translated notebook Part 09 - Intro to Encrypted Programs. * modify bengali translated notebook Part 10: Federated Learning with Encrypted Gradient Aggregation. * modify bengali translated notebook Part 08 bis - Introduction to Protocols. * modify bengali translation Part - 05. * remove Part 08 bis - Introduction to Protocols.ipynb from bengali translation. * modify bengal translation Part 09 - Intro to Encrypted Programs.ipynb notebook. Co-authored-by: Karl Higley <kmhigley@gmail.com>
… on Encrypted Data.ipynb (OpenMined#3445)
… Keras and TFE - Public Training.ipynb (OpenMined#3449)
…Keras and TFE - Secure Model Serving.ipynb (OpenMined#3466)
…Mined#3473) * Store the message log in `BaseWorker` in an unserialized format This bypasses some issues where serialization/deserialization changes the state or type of the objects passed to the serdes. That also needs to be fixed, but this is a smaller and still sensible change in the mean time. * Fix comment and formatting
…penMined#3477) * Remove `contents` methods (previously deprecated, partially removed) * Remove stray `contents` check in serde tests
* Add async send_command to websocker client * Update syft/workers/websocket_client.py Co-authored-by: S P Sharan <spsharan2000@gmail.com> * Update base.py * Make print statement only for verbose Co-authored-by: S P Sharan <spsharan2000@gmail.com>
The verbose output makes it harder to see where the problems are, because it includes files that already have the correct formatting. That makes it a needle/haystack problem trying to figure out what to change in order for the linting to pass.
…tions` (OpenMined#3465) * Add a test for registering `ComputationActions` in `Roles` * Add a target tensor to the `ComputationAction` test * Add a test for registering a `CommunicationAction` * Add an assertion to `return_ids` to `ComputationAction` test * Update method name in `ComputationAction` test * Update method name in `CommunicationAction` test * Add a check for return_ids in `CommunicationAction` test * Make the `CommunicationAction` registration test pass * Update `CommunicationAction` serialization * Update the surrounding code to use new `CommunicationAction` attrs * Fix the `make_communication_action` serde testing helper w/ new fields * Move shared fields to `Action` abstract base class and update `__init__`s * Move code to produce string representations into `Action` base class * Move `__eq__` into `Action` base class * Move `simplify` and `detail` to abstract `Action` class * Fix remaining issue with msgpack serialization * Move `bufferize` and `unbufferize` to abstract `Action` class * Improve error messages in the `Action` serde test helpers * Add annotations and type checking to `Action.__init__()` to sort out types for serialization * Fix upstream types that feed into `Actions` * Add type annotation to `Action.return_ids` * Take ownership of any objects stored locally * Point `syft-proto` dependency at PR branch * Add a test for taking ownership of objects added to object storage * Add simple `ComputationAction` tests * Remove unnecessary/untested support for lists/dicts as `Action` args * Point `syft-proto` at newly released version * Simplify `un/bufferize_args` by using list comprehensions
* remove owner from placeholder constructor args * interact with roles instead of inputs for Protocols * remove TODO comment * fix serde tests with new syntax
* Sketch out a comms tracing test * try adding ph.send * Apply `Placeholder` conversion from `handle_func_command` in `send` * Make the `Protocol` test pass again (after rebase) * add more tests 5 pass, 4 fails * formatting * formatting * lint * add test * remove test * get test working * remote_get working * 9 tests working * doc string + clean up * fix empty share test + more doc strings * fix empty share test + more doc strings * fix empty share test + more doc strings * Remove a stray change Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: maddie_shang <mshang@tableau.com>
…ined#3487) * syft: refactor BaseWorker.send_command to accept explicit arguments. syft: remove unused variables * syft: fix linter issues. * syft: fix argument type of args_ and update docstring for send_command of BaseWoker class. * syft, pointers: fix int type to tuple conversion. Co-authored-by: Shubham Gupta <shubamgupta3121@gmail.com>
* add reset method on Role * fix reset so that it doesn't erase state placeholders
…t. (OpenMined#3296) Working on issue OpenMined#3184, improving Plans flexibility. * Adding support for lists, dicts and tuples as input for Plans. * Adding a typecheck when calling a Plan, comparing the call arguments with the built arguments types. * New error when using different length nested structures or input arguments on a Plan at call time, compared to build time. * New error when using dicts at call that have different keys than the dicts with whom the Plan was built. * Added protobuf support for types. * Refactoring the type checking in a new class - NestedTypeWrapper. Co-authored-by: tudor <Tudor Cebere>
… Keras and TFE - Private Prediction Client.ipynb (OpenMined#3500) Error fixes in Bengali translation
This catches insufficient docstring issues up front instead of after 30 minutes of test execution time and cleans up a few of the build steps along the way.
Co-authored-by: Karl Higley <kmhigley@gmail.com>
Adding BaseDataset protobuffer suport. Co-authored-by: Karl Higley <kmhigley@gmail.com>
…penMined#3655) This renames `chain_call` to `map_chain_call` and introduces a method called `reduce_chain_call`. `map_chain_call` still gathers the values returned from each hook method that's called on a tensor chain, resulting in a list of values. `reduce_chain_call` passes the output value from each hook method into the next one, resulting in a single value.
* ADding support in msgpack for external libs. * Bug repair and proper testing. * reformat * proper_testing * reformat * removing import * small fix
* Extract `MessageHandler` from `BaseWorker` * Extract `AbstractMessageHandler` from `BaseMessageHandler` * Fix `ObjectStore` import * Fix assertion in `BaseMessageHandler.execute_computation_action()` * Add the ability to support multiple message handlers in `BaseWorker` * Update `chain_call` method to `map_chain_call` after merge * Fix owner attr in `BaseMessageHandler.execution_communication_action()`
…Crypten (with jails) (OpenMined#3565) * removing local party * parallelize virtual workers * demo training NN using syft + crypten (with jails) * unhook MPCTensor * Remove the hook from Crypten MPC Tensor (OpenMined#3619) Co-authored-by: Ayoub Benaissa <ayouben9@gmail.com> * CrypTen test coverage (OpenMined#3623) * fix jail + tests * ignore undef model name * cover jail with tests * test run_party * test hooking * disable ttp (not used anymore) we should think about how to provide this feature * refactor run_multiworkers * test failure of run_multiworkers * don't cover function that runs in diff process Co-authored-by: George-Cristian Muraru <murarugeorgec@gmail.com>
Merge master into crypten
* Fix plans framework * CrypTen Message Handler * Test fixup * Black fixup * Remove unused/undefined functions * Move message registration to worker level * Lint * Use global msgpack * factorize worker_id to rank translation * fix sequential model with last version of crypten Co-authored-by: youben11 <ayouben9@gmail.com>
* detect duplicate ids duplicate ids create issues in translation between worker_ids and ranks * tests * lint * no list comprehension needed * noqa
* new strategy for storing rank to id translation tables Each CrypTen computation is identified using a cid, it might be different in different workers. Translation tables are indexed using CIDs. Global table and the CID of the running computation are stored under syft.frameworks.crypten, we assume that parties are separate processes and that CID is overwritten after process creation, which means that even if parties are running in the same machine, they won't manipulate the global CID concurrently. * user friendly API for getting worker using ranks * remove old sy.local_worker.rank_to_worker_id * typo * comments to explain RANK_TO_WORKER_ID * use syft's id_provider * lint
gmuraru
force-pushed
the
gm-employ-onnx-models
branch
2 times, most recently
from
808c872 to
1accdd7
Compare
* parent db61f12 author George Muraru <murarugeorgec@gmail.com> 1589458543 +0300 committer George Muraru <murarugeorgec@gmail.com> 1592429175 +0300 Initial POC for models in plans * Format linter * Flake * Add tests for OnnxModel * Add decrypt as method to overload * Add tests compared to jail * Review suggestions * Remove protobuf for Onnx and linter check * Remove TODO indent * Put wrap back * Remove OnnxModel from tests * Remove duplicated test * Put comment back * Flake8 fix * Fix test serde * Add tests to increase cov * Changes from crypten - prevent enc of shapes * Add no coverage * Remove Onnx Model * Fix syft-proto * Fix review * Fix review some more
gmuraru
force-pushed
the
gm-employ-onnx-models
branch
3 times, most recently
from
10fcc7b to
08c6f36
Compare
gmuraru
force-pushed
the
gm-employ-onnx-models
branch
from
08c6f36 to
bafc516
Compare
Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Use the OnnxModel that represents a new object that a worker can hold.
This object is a serialized Torch model and it will be used to load a crypten model (not encrypted).
In this scenario, the party that started the computation will have no idea about what the architecture of the
remote model is. It knows only that Bob and Alice has the model and it could run inference on some data and
get the response back.
Checklist: