only Flush once for the same stream in copyInputAcrossDevice() (#17303)

### Description
<!-- Describe your changes. -->
In CopyInputAcrossDevice() function, we assign each feed a stream to
copy across device, once the copy is done, each stream will trigger the
Flush() function which is undesired. Same stream should be only flushed
once


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
This change is to address a perf issue of TLNGv4 inference which
contains subgraph with many input feeds.
This commit is contained in:
cao lei 2023-08-30 16:10:26 -07:00 committed by GitHub
parent 70e8c23944
commit 64f06d0b4a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23

View file

@ -478,9 +478,9 @@ static common::Status CopyInputsAcrossDevices(const SessionState& session_state,
// TODO: this sync is because the graph inputs can be consumed by multiple stream,
// but we can only place the MemCpyAsync on one of the stream. Ideally we should make
// other stream wait on the event of the memory copy stream, instead of host sync stream.
std::unordered_set<Stream*> visited;
for (auto* stream : feed_streams) {
if (stream)
stream->Flush();
if (stream && visited.insert(stream).second) stream->Flush();
}
return Status::OK();
}