pytorch/scripts/xcode_build.rb

80 lines
2.7 KiB
Ruby
Raw Normal View History

require 'optparse'
require 'xcodeproj'
options = {}
option_parser = OptionParser.new do |opts|
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
opts.banner = 'Tools for building PyTorch iOS framework on MacOS'
opts.on('-i', '--install_path ', 'path to the cmake install folder') { |value|
options[:install] = value
}
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
opts.on('-x', '--xcodeproj_path ', 'path to the XCode project file') { |value|
options[:xcodeproj] = value
}
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
opts.on('-p', '--platform ', 'platform for the current build, OS or SIMULATOR') { |value|
options[:platform] = value
}
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
opts.on('-c', '--provisioning_profile ', 'provisioning profile for code signing') { |value|
options[:profile] = value
}
opts.on('-t', '--team_id ', 'development team ID') { |value|
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
options[:team_id] = value
}
end.parse!
puts options.inspect
install_path = File.expand_path(options[:install])
if not Dir.exist? (install_path)
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
raise "path don't exist:#{install_path}!"
end
xcodeproj_path = File.expand_path(options[:xcodeproj])
if not File.exist? (xcodeproj_path)
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
raise "path don't exist:#{xcodeproj_path}!"
end
project = Xcodeproj::Project.open(xcodeproj_path)
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
target = project.targets.first #TestApp
header_search_path = ['$(inherited)', "#{install_path}/include"]
libraries_search_path = ['$(inherited)', "#{install_path}/lib"]
other_linker_flags = ['$(inherited)', "-all_load"]
target.build_configurations.each do |config|
config.build_settings['HEADER_SEARCH_PATHS'] = header_search_path
config.build_settings['LIBRARY_SEARCH_PATHS'] = libraries_search_path
config.build_settings['OTHER_LDFLAGS'] = other_linker_flags
config.build_settings['ENABLE_BITCODE'] = 'No'
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
dev_team_id = options[:team_id]
if not dev_team_id
raise "Please sepecify a valid development team id for code signing"
end
config.build_settings['DEVELOPMENT_TEAM'] = dev_team_id
end
# link static libraries
target.frameworks_build_phases.clear
Split libtorch.so back into libtorch_{cpu,cuda,hip} (#30315) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/30315 The new structure is that libtorch_cpu contains the bulk of our code, and libtorch depends on libtorch_cpu and libtorch_cuda. This is a reland of https://github.com/pytorch/pytorch/pull/29731 but I've extracted all of the prep work into separate PRs which can be landed before this one. Some things of note: * torch/csrc/cuda/nccl.cpp was added to the wrong list of SRCS, now fixed (this didn't matter before because previously they were all in the same library) * The dummy file for libtorch was brought back from the dead; it was previously deleted in #20774 In an initial version of the patch, I forgot to make torch_cuda explicitly depend on torch_cpu. This lead to some very odd errors, most notably "bin/blob_test: hidden symbol `_ZNK6google8protobuf5Arena17OnArenaAllocationEPKSt9type_infom' in lib/libprotobuf.a(arena.cc.o) is referenced by DSO" * A number of places in Android/iOS builds have to add torch_cuda explicitly as a library, as they do not have transitive dependency calculation working correctly * I had to torch_cpu/torch_cuda caffe2_interface_library so that they get whole-archived linked into torch when you statically link. And I had to do this in an *exported* fashion because torch needs to depend on torch_cpu_library. In the end I exported everything and removed the redefinition in the Caffe2Config.cmake. However, I am not too sure why the old code did it in this way in the first place; however, it doesn't seem to have broken anything to switch it this way. * There's some uses of `__HIP_PLATFORM_HCC__` still in `torch_cpu` code, so I had to apply it to that library too (UGH). This manifests as a failer when trying to run the CUDA fuser. This doesn't really matter substantively right now because we still in-place HIPify, but it would be good to fix eventually. This was a bit difficult to debug because of an unrelated HIP bug, see https://github.com/ROCm-Developer-Tools/HIP/issues/1706 Fixes #27215 (as our libraries are smaller), and executes on part of the plan in #29235. Signed-off-by: Edward Z. Yang <ezyang@fb.com> Test Plan: Imported from OSS Differential Revision: D18790941 Pulled By: ezyang fbshipit-source-id: 01296f6089d3de5e8365251b490c51e694f2d6c7
2019-12-04 16:03:10 +00:00
libs = ['libc10.a', 'libclog.a', 'libnnpack.a', 'libeigen_blas.a', 'libcpuinfo.a', 'libpytorch_qnnpack.a', 'libtorch_cpu.a', 'libtorch.a']
for lib in libs do
path = "#{install_path}/lib/#{lib}"
if File.exist?(path)
libref = project.frameworks_group.new_file(path)
target.frameworks_build_phases.add_file_reference(libref)
end
end
project.save
sdk = nil
if options[:platform] == 'SIMULATOR'
sdk = 'iphonesimulator'
elsif options[:platform] == 'OS'
sdk = 'iphoneos'
else
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
raise "unsupported platform #{options[:platform]}"
end
profile = options[:profile]
if not profile
raise "no provisioning profile found!"
end
# run xcodebuild
Install developer certificate for code signing (#27593) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/27593 ## Summary Since the nightly jobs are lack of testing phases, we don't really have a way to test the binary before uploading it to AWS. To make the work more solid, we need to figure out a way to verify the binary. Fortunately, the XCode tool chain offers a way to build your app without XCode app, which is the [xcodebuild](https://developer.apple.com/library/archive/technotes/tn2339/_index.html) command. Now we can link our binary to a testing app and run `xcodebuild` to to see if there is any linking error. The PRs below have already done some of the preparation jobs - [#26261](https://github.com/pytorch/pytorch/pull/26261) - [#26632](https://github.com/pytorch/pytorch/pull/26632) The challenge comes when testing the arm64 build as we don't have a way to code-sign our TestApp. Circle CI has a [tutorial](https://circleci.com/docs/2.0/ios-codesigning/) but is too complicated to implement. Anyway, I figured out an easier way to do it 1. Disable automatically code sign in XCode 2. Export the encoded developer certificate and provisioning profile to org-context in Circle CI (done) 3. Install the developer certificate to the key chain store on CI machines via Fastlane. 4. Add the testing code to PR jobs and verify the result. 5. Add the testing code to nightly jobs and verify the result. ## Test Plan - Both PR jobs and nightly jobs can finish successfully. - `xcodebuild` can finish successfully Test Plan: Imported from OSS Differential Revision: D17848814 Pulled By: xta0 fbshipit-source-id: 48353f001c38e61eed13a43943253cae30d8831a
2019-10-10 03:05:54 +00:00
exec "xcodebuild clean build -project #{xcodeproj_path} -target #{target.name} -sdk #{sdk} -configuration Release PROVISIONING_PROFILE_SPECIFIER=#{profile}"