pytorch/scripts/xcode_build.rb

80 lines
2.8 KiB
Ruby
Raw Normal View History

require 'optparse'
require 'xcodeproj'
options = {}
Mobile Backend: NHWC memory layout + XNNPACK integration. (#33722) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/33722 In order to improve CPU performance on floating-point models on mobile, this PR introduces a new CPU backend for mobile that implements the most common mobile operators with NHWC memory layout support through integration with XNNPACK. XNNPACK itself, and this codepath, are currently only included in the build, but the actual integration is gated with USE_XNNPACK preprocessor guards. This preprocessor symbol is intentionally not passed on to the compiler, so as to enable this rollout in multiple stages in follow up PRs. This changeset will build XNNPACK as part of the build if the identically named USE_XNNPACK CMAKE variable, defaulted to ON, is enabled, but will not actually expose or enable this code path in any other way. Furthermore, it is worth pointing out that in order to efficiently map models to these operators, some front-end method of exposing this backend to the user is needed. The less efficient implementation would be to hook these operators into their corresponding native implementations, granted that a series of XNNPACK-specific conditions are met, much like how NNPACK is integrated with PyTorch today for instance. Having said that, while the above implementation is still expected to outperform NNPACK based on the benchmarks I ran, the above integration would be leave a considerable gap between the performance achieved and the maximum performance potential XNNPACK enables, as it does not provide a way to compute and factor out one-time operations out of the inner most forward() loop. The more optimal solution, and one we will decide on soon, would involve either providing a JIT pass that maps nn operators onto these newly introduced operators, while allowing one-time calculations to be factored out, much like quantized mobile models. Alternatively, new eager-mode modules can also be introduced that would directly call into these implementations either through c10 or some other mechanism, also allowing for decoupling of op creation from op execution. This PR does not include any of the front end changes mentioned above. Neither does it include the mobile threadpool unification present in the original https://github.com/pytorch/pytorch/issues/30644. Furthermore, this codepath seems to be faster than NNPACK in a good number of use cases, which can potentially allow us to remove NNPACK from aten to make the codebase a little simpler, granted that there is widespread support for such a move. Regardless, these changes will be introduced gradually and in a more controlled way in subsequent PRs. Pull Request resolved: https://github.com/pytorch/pytorch/pull/32509 Test Plan: Build: CI Functionality: Not exposed Reviewed By: dreiss Differential Revision: D20069796 Pulled By: AshkanAliabadi fbshipit-source-id: d46c1c91d4bea91979ea5bd46971ced5417d309c
2020-02-25 05:53:34 +00:00
option_parser = OptionParser.new do |opts|
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
opts.banner = 'Tools for building PyTorch iOS framework on MacOS'
opts.on('-i', '--install_path ', 'path to the cmake install folder') { |value|
options[:install] = value
}
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
opts.on('-x', '--xcodeproj_path ', 'path to the XCode project file') { |value|
options[:xcodeproj] = value
}
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
opts.on('-p', '--platform ', 'platform for the current build, OS or SIMULATOR') { |value|
options[:platform] = value
}
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
opts.on('-c', '--provisioning_profile ', 'provisioning profile for code signing') { |value|
options[:profile] = value
}
opts.on('-t', '--team_id ', 'development team ID') { |value|
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
options[:team_id] = value
}
end.parse!
puts options.inspect
install_path = File.expand_path(options[:install])
Mobile Backend: NHWC memory layout + XNNPACK integration. (#33722) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/33722 In order to improve CPU performance on floating-point models on mobile, this PR introduces a new CPU backend for mobile that implements the most common mobile operators with NHWC memory layout support through integration with XNNPACK. XNNPACK itself, and this codepath, are currently only included in the build, but the actual integration is gated with USE_XNNPACK preprocessor guards. This preprocessor symbol is intentionally not passed on to the compiler, so as to enable this rollout in multiple stages in follow up PRs. This changeset will build XNNPACK as part of the build if the identically named USE_XNNPACK CMAKE variable, defaulted to ON, is enabled, but will not actually expose or enable this code path in any other way. Furthermore, it is worth pointing out that in order to efficiently map models to these operators, some front-end method of exposing this backend to the user is needed. The less efficient implementation would be to hook these operators into their corresponding native implementations, granted that a series of XNNPACK-specific conditions are met, much like how NNPACK is integrated with PyTorch today for instance. Having said that, while the above implementation is still expected to outperform NNPACK based on the benchmarks I ran, the above integration would be leave a considerable gap between the performance achieved and the maximum performance potential XNNPACK enables, as it does not provide a way to compute and factor out one-time operations out of the inner most forward() loop. The more optimal solution, and one we will decide on soon, would involve either providing a JIT pass that maps nn operators onto these newly introduced operators, while allowing one-time calculations to be factored out, much like quantized mobile models. Alternatively, new eager-mode modules can also be introduced that would directly call into these implementations either through c10 or some other mechanism, also allowing for decoupling of op creation from op execution. This PR does not include any of the front end changes mentioned above. Neither does it include the mobile threadpool unification present in the original https://github.com/pytorch/pytorch/issues/30644. Furthermore, this codepath seems to be faster than NNPACK in a good number of use cases, which can potentially allow us to remove NNPACK from aten to make the codebase a little simpler, granted that there is widespread support for such a move. Regardless, these changes will be introduced gradually and in a more controlled way in subsequent PRs. Pull Request resolved: https://github.com/pytorch/pytorch/pull/32509 Test Plan: Build: CI Functionality: Not exposed Reviewed By: dreiss Differential Revision: D20069796 Pulled By: AshkanAliabadi fbshipit-source-id: d46c1c91d4bea91979ea5bd46971ced5417d309c
2020-02-25 05:53:34 +00:00
if not Dir.exist? (install_path)
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
raise "path don't exist:#{install_path}!"
end
xcodeproj_path = File.expand_path(options[:xcodeproj])
Mobile Backend: NHWC memory layout + XNNPACK integration. (#33722) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/33722 In order to improve CPU performance on floating-point models on mobile, this PR introduces a new CPU backend for mobile that implements the most common mobile operators with NHWC memory layout support through integration with XNNPACK. XNNPACK itself, and this codepath, are currently only included in the build, but the actual integration is gated with USE_XNNPACK preprocessor guards. This preprocessor symbol is intentionally not passed on to the compiler, so as to enable this rollout in multiple stages in follow up PRs. This changeset will build XNNPACK as part of the build if the identically named USE_XNNPACK CMAKE variable, defaulted to ON, is enabled, but will not actually expose or enable this code path in any other way. Furthermore, it is worth pointing out that in order to efficiently map models to these operators, some front-end method of exposing this backend to the user is needed. The less efficient implementation would be to hook these operators into their corresponding native implementations, granted that a series of XNNPACK-specific conditions are met, much like how NNPACK is integrated with PyTorch today for instance. Having said that, while the above implementation is still expected to outperform NNPACK based on the benchmarks I ran, the above integration would be leave a considerable gap between the performance achieved and the maximum performance potential XNNPACK enables, as it does not provide a way to compute and factor out one-time operations out of the inner most forward() loop. The more optimal solution, and one we will decide on soon, would involve either providing a JIT pass that maps nn operators onto these newly introduced operators, while allowing one-time calculations to be factored out, much like quantized mobile models. Alternatively, new eager-mode modules can also be introduced that would directly call into these implementations either through c10 or some other mechanism, also allowing for decoupling of op creation from op execution. This PR does not include any of the front end changes mentioned above. Neither does it include the mobile threadpool unification present in the original https://github.com/pytorch/pytorch/issues/30644. Furthermore, this codepath seems to be faster than NNPACK in a good number of use cases, which can potentially allow us to remove NNPACK from aten to make the codebase a little simpler, granted that there is widespread support for such a move. Regardless, these changes will be introduced gradually and in a more controlled way in subsequent PRs. Pull Request resolved: https://github.com/pytorch/pytorch/pull/32509 Test Plan: Build: CI Functionality: Not exposed Reviewed By: dreiss Differential Revision: D20069796 Pulled By: AshkanAliabadi fbshipit-source-id: d46c1c91d4bea91979ea5bd46971ced5417d309c
2020-02-25 05:53:34 +00:00
if not File.exist? (xcodeproj_path)
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
raise "path don't exist:#{xcodeproj_path}!"
end
project = Xcodeproj::Project.open(xcodeproj_path)
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
target = project.targets.first #TestApp
header_search_path = ['$(inherited)', "#{install_path}/include"]
libraries_search_path = ['$(inherited)', "#{install_path}/lib"]
other_linker_flags = ['$(inherited)', "-all_load"]
target.build_configurations.each do |config|
config.build_settings['HEADER_SEARCH_PATHS'] = header_search_path
config.build_settings['LIBRARY_SEARCH_PATHS'] = libraries_search_path
config.build_settings['OTHER_LDFLAGS'] = other_linker_flags
config.build_settings['ENABLE_BITCODE'] = 'No'
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
dev_team_id = options[:team_id]
if not dev_team_id and options[:platform] == 'OS'
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
raise "Please sepecify a valid development team id for code signing"
end
config.build_settings['DEVELOPMENT_TEAM'] = dev_team_id
end
# link static libraries
target.frameworks_build_phases.clear
libs = ['libc10.a', 'libclog.a', 'libpthreadpool.a', 'libXNNPACK.a', 'libeigen_blas.a', 'libcpuinfo.a', 'libpytorch_qnnpack.a', 'libtorch_cpu.a', 'libtorch.a']
Mobile Backend: NHWC memory layout + XNNPACK integration. (#33722) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/33722 In order to improve CPU performance on floating-point models on mobile, this PR introduces a new CPU backend for mobile that implements the most common mobile operators with NHWC memory layout support through integration with XNNPACK. XNNPACK itself, and this codepath, are currently only included in the build, but the actual integration is gated with USE_XNNPACK preprocessor guards. This preprocessor symbol is intentionally not passed on to the compiler, so as to enable this rollout in multiple stages in follow up PRs. This changeset will build XNNPACK as part of the build if the identically named USE_XNNPACK CMAKE variable, defaulted to ON, is enabled, but will not actually expose or enable this code path in any other way. Furthermore, it is worth pointing out that in order to efficiently map models to these operators, some front-end method of exposing this backend to the user is needed. The less efficient implementation would be to hook these operators into their corresponding native implementations, granted that a series of XNNPACK-specific conditions are met, much like how NNPACK is integrated with PyTorch today for instance. Having said that, while the above implementation is still expected to outperform NNPACK based on the benchmarks I ran, the above integration would be leave a considerable gap between the performance achieved and the maximum performance potential XNNPACK enables, as it does not provide a way to compute and factor out one-time operations out of the inner most forward() loop. The more optimal solution, and one we will decide on soon, would involve either providing a JIT pass that maps nn operators onto these newly introduced operators, while allowing one-time calculations to be factored out, much like quantized mobile models. Alternatively, new eager-mode modules can also be introduced that would directly call into these implementations either through c10 or some other mechanism, also allowing for decoupling of op creation from op execution. This PR does not include any of the front end changes mentioned above. Neither does it include the mobile threadpool unification present in the original https://github.com/pytorch/pytorch/issues/30644. Furthermore, this codepath seems to be faster than NNPACK in a good number of use cases, which can potentially allow us to remove NNPACK from aten to make the codebase a little simpler, granted that there is widespread support for such a move. Regardless, these changes will be introduced gradually and in a more controlled way in subsequent PRs. Pull Request resolved: https://github.com/pytorch/pytorch/pull/32509 Test Plan: Build: CI Functionality: Not exposed Reviewed By: dreiss Differential Revision: D20069796 Pulled By: AshkanAliabadi fbshipit-source-id: d46c1c91d4bea91979ea5bd46971ced5417d309c
2020-02-25 05:53:34 +00:00
for lib in libs do
path = "#{install_path}/lib/#{lib}"
if File.exist?(path)
libref = project.frameworks_group.new_file(path)
target.frameworks_build_phases.add_file_reference(libref)
end
end
project.save
sdk = nil
if options[:platform] == 'SIMULATOR'
sdk = 'iphonesimulator'
elsif options[:platform] == 'OS'
sdk = 'iphoneos'
else
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
raise "unsupported platform #{options[:platform]}"
Mobile Backend: NHWC memory layout + XNNPACK integration. (#33722) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/33722 In order to improve CPU performance on floating-point models on mobile, this PR introduces a new CPU backend for mobile that implements the most common mobile operators with NHWC memory layout support through integration with XNNPACK. XNNPACK itself, and this codepath, are currently only included in the build, but the actual integration is gated with USE_XNNPACK preprocessor guards. This preprocessor symbol is intentionally not passed on to the compiler, so as to enable this rollout in multiple stages in follow up PRs. This changeset will build XNNPACK as part of the build if the identically named USE_XNNPACK CMAKE variable, defaulted to ON, is enabled, but will not actually expose or enable this code path in any other way. Furthermore, it is worth pointing out that in order to efficiently map models to these operators, some front-end method of exposing this backend to the user is needed. The less efficient implementation would be to hook these operators into their corresponding native implementations, granted that a series of XNNPACK-specific conditions are met, much like how NNPACK is integrated with PyTorch today for instance. Having said that, while the above implementation is still expected to outperform NNPACK based on the benchmarks I ran, the above integration would be leave a considerable gap between the performance achieved and the maximum performance potential XNNPACK enables, as it does not provide a way to compute and factor out one-time operations out of the inner most forward() loop. The more optimal solution, and one we will decide on soon, would involve either providing a JIT pass that maps nn operators onto these newly introduced operators, while allowing one-time calculations to be factored out, much like quantized mobile models. Alternatively, new eager-mode modules can also be introduced that would directly call into these implementations either through c10 or some other mechanism, also allowing for decoupling of op creation from op execution. This PR does not include any of the front end changes mentioned above. Neither does it include the mobile threadpool unification present in the original https://github.com/pytorch/pytorch/issues/30644. Furthermore, this codepath seems to be faster than NNPACK in a good number of use cases, which can potentially allow us to remove NNPACK from aten to make the codebase a little simpler, granted that there is widespread support for such a move. Regardless, these changes will be introduced gradually and in a more controlled way in subsequent PRs. Pull Request resolved: https://github.com/pytorch/pytorch/pull/32509 Test Plan: Build: CI Functionality: Not exposed Reviewed By: dreiss Differential Revision: D20069796 Pulled By: AshkanAliabadi fbshipit-source-id: d46c1c91d4bea91979ea5bd46971ced5417d309c
2020-02-25 05:53:34 +00:00
end
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
profile = options[:profile]
if not profile and options[:platform] == 'OS'
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
raise "no provisioning profile found!"
Mobile Backend: NHWC memory layout + XNNPACK integration. (#33722) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/33722 In order to improve CPU performance on floating-point models on mobile, this PR introduces a new CPU backend for mobile that implements the most common mobile operators with NHWC memory layout support through integration with XNNPACK. XNNPACK itself, and this codepath, are currently only included in the build, but the actual integration is gated with USE_XNNPACK preprocessor guards. This preprocessor symbol is intentionally not passed on to the compiler, so as to enable this rollout in multiple stages in follow up PRs. This changeset will build XNNPACK as part of the build if the identically named USE_XNNPACK CMAKE variable, defaulted to ON, is enabled, but will not actually expose or enable this code path in any other way. Furthermore, it is worth pointing out that in order to efficiently map models to these operators, some front-end method of exposing this backend to the user is needed. The less efficient implementation would be to hook these operators into their corresponding native implementations, granted that a series of XNNPACK-specific conditions are met, much like how NNPACK is integrated with PyTorch today for instance. Having said that, while the above implementation is still expected to outperform NNPACK based on the benchmarks I ran, the above integration would be leave a considerable gap between the performance achieved and the maximum performance potential XNNPACK enables, as it does not provide a way to compute and factor out one-time operations out of the inner most forward() loop. The more optimal solution, and one we will decide on soon, would involve either providing a JIT pass that maps nn operators onto these newly introduced operators, while allowing one-time calculations to be factored out, much like quantized mobile models. Alternatively, new eager-mode modules can also be introduced that would directly call into these implementations either through c10 or some other mechanism, also allowing for decoupling of op creation from op execution. This PR does not include any of the front end changes mentioned above. Neither does it include the mobile threadpool unification present in the original https://github.com/pytorch/pytorch/issues/30644. Furthermore, this codepath seems to be faster than NNPACK in a good number of use cases, which can potentially allow us to remove NNPACK from aten to make the codebase a little simpler, granted that there is widespread support for such a move. Regardless, these changes will be introduced gradually and in a more controlled way in subsequent PRs. Pull Request resolved: https://github.com/pytorch/pytorch/pull/32509 Test Plan: Build: CI Functionality: Not exposed Reviewed By: dreiss Differential Revision: D20069796 Pulled By: AshkanAliabadi fbshipit-source-id: d46c1c91d4bea91979ea5bd46971ced5417d309c
2020-02-25 05:53:34 +00:00
end
# run xcodebuild
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
exec "xcodebuild clean build -project #{xcodeproj_path} -target #{target.name} -sdk #{sdk} -configuration Release PROVISIONING_PROFILE_SPECIFIER=#{profile}"