Skip to content

Conversation

@rnwang04
Copy link
Contributor

@rnwang04 rnwang04 commented Oct 10, 2024

Description

Initial integration of NPU level0 pipeline model.

1. Why the change?

To integrate NPU level0 pipeline model into ipex-llm.

2. User API changes

See example usage.

3. Summary of the change

  • need to add pipeline.dll into libs, which need to update release workflow [TODO]
  • Implement cpp bindings to load necessary functions like generate_serve and InitLLMPipeline from pipeline.dll
  • Create a new Class to support such NPU level0 pipeline model
  • Implement initial from_pretrained method to load blobs and init llm pipeline
  • Implement initial generate method to start serving using threading.Thread and mock a generate process
  • Add an initial example for Llama2 NPU Pipeline model

4. How to test?

  • Unit test: Please manually trigger the PR Validation here by inputting the PR number (e.g., 1234). And paste your action link here once it has been successfully finished.
  • Application test

@rnwang04 rnwang04 requested a review from jason-dai October 10, 2024 08:51
@rnwang04 rnwang04 changed the title Npu level0 Initial support of NPU level0 Model Oct 10, 2024
Copy link
Contributor

@jason-dai jason-dai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rnwang04
Copy link
Contributor Author

@rnwang04 rnwang04 merged commit 4d93bb8 into intel:main Oct 11, 2024
1 check passed
@rnwang04 rnwang04 deleted the npu-level0 branch October 11, 2024 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants