Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .cspell-wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -215,3 +215,11 @@ MATEUSZ
BLAZEFACE
Blazeface
blazeface
webfetch
prebuild
embedders
upsamples
artefacts
categorisation
chipmunked
autoregressive
15 changes: 15 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,18 @@ jobs:

- name: Build all packages
run: yarn workspaces foreach --all --topological-dev run prepare

test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6

- name: Setup
uses: ./.github/actions/setup

- name: Typecheck test files
run: yarn workspace react-native-executorch typecheck:tests

- name: Run API contract tests
run: yarn workspace react-native-executorch test --ci
2 changes: 1 addition & 1 deletion apps/computer-vision/app/text_to_image/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ export default function TextToImageScreen() {

try {
const start = Date.now();
const output = await model.generate(input, imageSize, steps);
const output = await model.forward(input, imageSize, steps);

if (output.length) {
setImage(output);
Expand Down
4 changes: 3 additions & 1 deletion apps/speech/screens/Quiz.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,9 @@ const createAudioBufferFromVector = (

export const Quiz = ({ onBack }: { onBack: () => void }) => {
// --- Hooks & State ---
const model = useTextToSpeech(models.text_to_speech.kokoro.en_us.santa());
const model = useTextToSpeech({
model: models.text_to_speech.kokoro.en_us.santa(),
});

const [shuffledQuestions] = useState(() => shuffleArray(QUESTIONS));
const [currentIndex, setCurrentIndex] = useState(0);
Expand Down
4 changes: 3 additions & 1 deletion apps/speech/screens/TextToSpeechLLMScreen.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@ export const TextToSpeechLLMScreen = ({ onBack }: TextToSpeechLLMProps) => {
const [displayText, setDisplayText] = useState('');
const [isTtsStreaming, setIsTtsStreaming] = useState(false);
const llm = useLLM({ model: models.llm.llama3_2_1b() });
const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
const tts = useTextToSpeech({
model: models.text_to_speech.kokoro.en_us.heart(),
});

const processedLengthRef = useRef(0);
const audioContextRef = useRef<AudioContext | null>(null);
Expand Down
2 changes: 1 addition & 1 deletion apps/speech/screens/TextToSpeechScreen.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ export const TextToSpeechScreen = ({ onBack }: { onBack: () => void }) => {
const [selectedSpeaker, setSelectedSpeaker] =
useState<TextToSpeechModelConfig>(tts.en_us.heart());

const model = useTextToSpeech(selectedSpeaker);
const model = useTextToSpeech({ model: selectedSpeaker });

const [inputText, setInputText] = useState('');
const [isPlaying, setIsPlaying] = useState(false);
Expand Down
6 changes: 3 additions & 3 deletions docs/docs/01-fundamentals/02-loading-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ initExecutorch({
### Load from React Native assets folder (for files < 512MB)

```typescript
useExecutorchModule({
useExecutorch({
modelSource: require('../assets/lfm2_5.pte'),
});
```
Expand All @@ -100,7 +100,7 @@ useExecutorchModule({
For files larger than 512MB or when you want to keep size of the app smaller, you can load the model from a remote URL (e.g. HuggingFace).

```typescript
useExecutorchModule({
useExecutorch({
modelSource: 'https://.../lfm2_5.pte',
});
```
Expand All @@ -110,7 +110,7 @@ useExecutorchModule({
If you prefer to delegate the process of obtaining and loading model and tokenizer files to the user, you can use the following method:

```typescript
useExecutorchModule({
useExecutorch({
modelSource: 'file:///var/mobile/.../lfm2_5.pte',
});
```
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/01-fundamentals/03-frequently-asked-questions.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Each hook documentation subpage (useClassification, useLLM, etc.) contains a sup

### How can I run my own AI model?

To run your own model, you need to directly access the underlying [ExecuTorch Module API](https://pytorch.org/executorch/stable/extension-module.html). We provide [React hook](../03-hooks/03-executorch-bindings/useExecutorchModule.md) along with a [TypeScript alternative](../04-typescript-api/03-executorch-bindings/ExecutorchModule.md), which serve as a way to use the aforementioned API without the need of diving into native code. In order to get a model in a format runnable by the runtime, you'll need to get your hands dirty with some ExecuTorch knowledge. For more guides on exporting models, please refer to the [ExecuTorch tutorials](https://pytorch.org/executorch/stable/tutorials/export-to-executorch-tutorial.html). Once you obtain your model in a `.pte` format, you can run it with `useExecuTorchModule` and `ExecuTorchModule`.
To run your own model, you need to directly access the underlying [ExecuTorch Module API](https://pytorch.org/executorch/stable/extension-module.html). We provide [React hook](../03-hooks/03-executorch-bindings/useExecutorch.md) along with a [TypeScript alternative](../04-typescript-api/03-executorch-bindings/ExecutorchModule.md), which serve as a way to use the aforementioned API without the need of diving into native code. In order to get a model in a format runnable by the runtime, you'll need to get your hands dirty with some ExecuTorch knowledge. For more guides on exporting models, please refer to the [ExecuTorch tutorials](https://pytorch.org/executorch/stable/tutorials/export-to-executorch-tutorial.html). Once you obtain your model in a `.pte` format, you can run it with `useExecutorch` and `ExecutorchModule`.

### How React Native ExecuTorch works under the hood?

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ You can play the generated waveform in any way most suitable to you; however, in
import { models, useTextToSpeech } from 'react-native-executorch';
import { AudioContext } from 'react-native-audio-api';

const model = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
const model = useTextToSpeech({
model: models.text_to_speech.kokoro.en_us.heart(),
});

const audioContext = new AudioContext({ sampleRate: 24000 });

Expand All @@ -56,15 +58,13 @@ const handleSpeech = async (text: string) => {

### Arguments

`useTextToSpeech` takes [`TextToSpeechModelConfig`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md) that consists of:
`useTextToSpeech` takes [`TextToSpeechProps`](../../06-api-reference/interfaces/TextToSpeechProps.md), an object containing:

- `model` of type [`TextToSpeechModelSources`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md) containing the [`durationPredictorSource`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#durationpredictorsource), [`synthesizerSource`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#synthesizersource), and [`modelName`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#modelname).
- [`voiceSource`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#voicesource) of type [`ResourceSource`](../../06-api-reference/type-aliases/ResourceSource.md) - configuration of specific voice used in TTS.
- [`phonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#phonemizerconfig) of type [`TextToSpeechPhonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechPhonemizerConfig.md) - configuration of the phonemizer.

`useTextToSpeech`'s second optional argument is an object with:

- `preventLoad` which prevents auto-loading of the model.
- `model` of type [`TextToSpeechModelConfig`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md), which itself consists of:
- [`model`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#model) of type [`TextToSpeechModelSources`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md) — bundles the [`durationPredictorSource`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#durationpredictorsource), [`synthesizerSource`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#synthesizersource), and [`modelName`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md#modelname).
- [`voiceSource`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#voicesource) of type [`ResourceSource`](../../06-api-reference/type-aliases/ResourceSource.md) — configuration of the specific voice used in TTS.
- [`phonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#phonemizerconfig) of type [`TextToSpeechPhonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechPhonemizerConfig.md) — configuration of the phonemizer.
- An optional flag `preventLoad` which prevents auto-loading of the model.

You need more details? Check the following resources:

Expand Down Expand Up @@ -115,7 +115,9 @@ import { models, useTextToSpeech } from 'react-native-executorch';
import { AudioContext } from 'react-native-audio-api';

export default function App() {
const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
const tts = useTextToSpeech({
model: models.text_to_speech.kokoro.en_us.heart(),
});

const generateAudio = async () => {
const audioData = await tts.forward({
Expand Down Expand Up @@ -150,7 +152,9 @@ import { models, useTextToSpeech } from 'react-native-executorch';
import { AudioContext } from 'react-native-audio-api';

export default function App() {
const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
const tts = useTextToSpeech({
model: models.text_to_speech.kokoro.en_us.heart(),
});

const contextRef = useRef(new AudioContext({ sampleRate: 24000 }));

Expand Down Expand Up @@ -192,7 +196,9 @@ import React from 'react';
import { Button, View } from 'react-native';
import { models, useTextToSpeech } from 'react-native-executorch';
export default function App() {
const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
const tts = useTextToSpeech({
model: models.text_to_speech.kokoro.en_us.heart(),
});

const synthesizePhonemes = async () => {
// Example phonemes for "Hello"
Expand Down
8 changes: 4 additions & 4 deletions docs/docs/03-hooks/02-computer-vision/useTextToImage.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ const model = useTextToImage({
const input = 'a castle';

try {
const image = await model.generate(input);
const image = await model.forward(input);
} catch (error) {
console.error(error);
}
Expand All @@ -52,7 +52,7 @@ You need more details? Check the following resources:

## Running the model

To run the model, you can use the [`generate`](../../06-api-reference/interfaces/TextToImageType.md#generate) method. It accepts four arguments: a text prompt describing the requested image, a size of the image in pixels, a number of denoising steps, and an optional seed value, which enables reproducibility of the results.
To run the model, you can use the [`forward`](../../06-api-reference/interfaces/TextToImageType.md#forward) method. It accepts four arguments: a text prompt describing the requested image, a size of the image in pixels, a number of denoising steps, and an optional seed value, which enables reproducibility of the results.

The image size must be a multiple of 32 due to the architecture of the U-Net and VAE models. The seed should be a positive integer.

Expand All @@ -76,13 +76,13 @@ function App() {
const numSteps = 25;

try {
image = await model.generate(input, imageSize, numSteps);
image = await model.forward(input, imageSize, numSteps);
} catch (error) {
console.error(error);
}
//...

// `generate` returns a `file://` URI to the PNG saved on disk.
// `forward` returns a `file://` URI to the PNG saved on disk.
return <Image source={{ uri: image }} />;
}
```
Expand Down
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
---
title: useExecutorchModule
title: useExecutorch
---

useExecutorchModule provides React Native bindings to the ExecuTorch [Module API](https://pytorch.org/executorch/stable/extension-module.html) directly from JavaScript.
useExecutorch provides React Native bindings to the ExecuTorch [Module API](https://pytorch.org/executorch/stable/extension-module.html) directly from JavaScript.

:::info
These bindings are primarily intended for custom model integration where no dedicated hook exists. If you are considering using a provided model, first verify whether a dedicated hook is available. Dedicated hooks simplify the implementation process by managing necessary pre and post-processing automatically. Utilizing these can save you effort and reduce complexity, ensuring you do not implement additional handling that is already covered.
:::

## API Reference

- For detailed API Reference for `useExecutorchModule` see: [`useExecutorchModule` API Reference](../../06-api-reference/functions/useExecutorchModule.md).
- For detailed API Reference for `useExecutorch` see: [`useExecutorch` API Reference](../../06-api-reference/functions/useExecutorch.md).

## Initializing ExecuTorch Module

You can initialize the ExecuTorch module in your JavaScript application using the `useExecutorchModule` hook. This hook facilitates the loading of models from the specified source and prepares them for use.
You can initialize the ExecuTorch module in your JavaScript application using the `useExecutorch` hook. This hook facilitates the loading of models from the specified source and prepares them for use.

```typescript
import { useExecutorchModule } from 'react-native-executorch';
const executorchModule = useExecutorchModule({
import { useExecutorch } from 'react-native-executorch';
const executorchModule = useExecutorch({
modelSource: require('../assets/models/model.pte'),
});
```
Expand All @@ -29,19 +29,19 @@ For more information on loading resources, take a look at [loading models](../..

### Arguments

`useExecutorchModule` takes [`ExecutorchModuleProps`](../../06-api-reference/interfaces/ExecutorchModuleProps.md) that consists of:
`useExecutorch` takes [`ExecutorchModuleProps`](../../06-api-reference/interfaces/ExecutorchModuleProps.md) that consists of:

- `model` containing [`modelSource`](../../06-api-reference/interfaces/ExecutorchModuleProps.md#modelsource).
- An optional flag [`preventLoad`](../../06-api-reference/interfaces/ExecutorchModuleProps.md#preventload) which prevents auto-loading of the model.

You need more details? Check the following resources:

- For detailed information about `useExecutorchModule` arguments check this section: [`useExecutorchModule` arguments](../../06-api-reference/functions/useExecutorchModule.md#parameters).
- For detailed information about `useExecutorch` arguments check this section: [`useExecutorch` arguments](../../06-api-reference/functions/useExecutorch.md#parameters).
- For more information on loading resources, take a look at [loading models](../../01-fundamentals/02-loading-models.md) page.

### Returns

`useExecutorchModule` returns an object called `ExecutorchModuleType` containing bunch of functions to interact with arbitrarily chosen models. To get more details please read: [`ExecutorchModuleType` API Reference](../../06-api-reference/interfaces/ExecutorchModuleType.md).
`useExecutorch` returns an object called `ExecutorchModuleType` containing bunch of functions to interact with arbitrarily chosen models. To get more details please read: [`ExecutorchModuleType` API Reference](../../06-api-reference/interfaces/ExecutorchModuleType.md).

## TensorPtr

Expand All @@ -62,13 +62,9 @@ This example demonstrates the integration and usage of the ExecuTorch bindings w
First, import the necessary functions from the `react-native-executorch` package and initialize the ExecuTorch module with the specified style transfer model.

```typescript
import {
models,
useExecutorchModule,
ScalarType,
} from 'react-native-executorch';
import { models, useExecutorch, ScalarType } from 'react-native-executorch';
// Initialize the executorch module with the predefined style transfer model.
const executorchModule = useExecutorchModule({
const executorchModule = useExecutorch({
modelSource: models.style_transfer.candy(),
});
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ title: ExecutorchModule
ExecutorchModule provides TypeScript bindings for the underlying ExecuTorch [Module API](https://pytorch.org/executorch/stable/extension-module.html).

:::tip
For React applications, consider using the [`useExecutorchModule`](../../03-hooks/03-executorch-bindings/useExecutorchModule.md) hook instead, which provides automatic state management, loading progress tracking, and cleanup on unmount.
For React applications, consider using the [`useExecutorch`](../../03-hooks/03-executorch-bindings/useExecutorch.md) hook instead, which provides automatic state management, loading progress tracking, and cleanup on unmount.
:::

## API Reference
Expand All @@ -23,11 +23,10 @@ const inputTensor = {
scalarType: ScalarType.FLOAT,
};

// Creating an instance
const model = new ExecutorchModule();

// Loading the model
await model.load(models.style_transfer.candy());
// Creating and loading the model in a single step
const model = await ExecutorchModule.fromModelSource(
models.style_transfer.candy()
);

// Running the forward method
const output = await model.forward([inputTensor]);
Expand Down Expand Up @@ -57,13 +56,13 @@ First, import the necessary functions from the `react-native-executorch` package

```typescript
import { models, ExecutorchModule, ScalarType } from 'react-native-executorch';
// Initialize the executorch module
const executorchModule = new ExecutorchModule();

// Load the model with optional download progress callback
await executorchModule.load(models.style_transfer.candy(), (progress) => {
console.log(`Download progress: ${progress}%`);
});
// Initialize and load the executorch module with optional download progress callback.
const executorchModule = await ExecutorchModule.fromModelSource(
models.style_transfer.candy(),
(progress) => {
console.log(`Download progress: ${progress}%`);
}
);
```

### Setting up input parameters
Expand Down
4 changes: 3 additions & 1 deletion docs/docs/05-utilities/model-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,9 @@ const styled = useStyleTransfer({
```typescript
import { models, useTextToSpeech } from 'react-native-executorch';

const tts = useTextToSpeech(models.text_to_speech.kokoro.en_us.heart());
const tts = useTextToSpeech({
model: models.text_to_speech.kokoro.en_us.heart(),
});
// Other languages:
// models.text_to_speech.kokoro.en_gb.emma()
// models.text_to_speech.kokoro.fr.siwis()
Expand Down
Loading
Loading