On this page
Overview
Owns provider-selectable chat and the caller-selected browser/native STT/TTS configuration, lifecycle, synthesis, transcription, and playback boundary.
- Artifact
AI.js· esm - Classification
public first party - Availability
Browser + native bridge + TWiN Cloud - Normalization
Complete mutable caller-owned browser speech authority, SDK-owned provider registration/replacement/disposal, explicit STT/TTS activation, normalized Whisper STT and Kokoro TTS execution with default capacities 1 and 4 respectively, explicit provider-neutral execution snapshots through status(role,{execution:true}), sticky readiness, canonical window.user readiness without compatibility-event object-identity admission or polling, exact ordered structural calls with required arguments.message, atomic nonblank all-ID tool-result sequencing, complete all-choice streaming/data validation, native Ollama structural adaptation, complete-response then all validated tool callbacks then completion ordering, observed async callbacks, Blob/File transcription, default repeated-formatting-mark removal from cloned outbound TTS input, explicit textFormat:'plain' preservation for complete synthesis/preparation while streaming cleanup remains automatic and caller/visible content stays exact, immediate exact-segment synthesis, per-call voice/speed capture, optional final-segment pause on the existing audio clock, optional per-call terminal playback completion, detached preparation with complete semantic DBOPFS audio reuse and same-owner pending request sharing; each preparation owns its pending activation interest, final-interest cancellation aborts that activation while other interests preserve it, complete cache hits start no model, and cancellation after readiness leaves the ready model loaded; independent prepared playback without a model load for cached audio, ordered audio-clock scheduling, playable audio, active-generation TTS operation-failure routing, mute, and cancellation are normalized; provider/model/runtime/voice policy remains caller-owned. Both speech roles default execution.device to auto: webnn-npu when navigator.ml.createContext is exposed, then webgpu when navigator.gpu is exposed, then wasm for CPU execution. Failed backend loads clean up the pool before trying fresh Workers with the same prepared model and dtype; explicit devices never fall back. STT capacity only accepts 1; TTS accepts 1 through 4. Execution snapshots expose requestedDevice, selectedDevice, maxConcurrentRequests, and activeRequestCount for both roles; selectedDevice is null while unloaded and names the successful upstream session backend request when loaded. It does not prove every operation used the physical NPU: WebNN unsupported operations may use WASM, and compatibility depends on the browser, driver, hardware, and graph. High-level LLM selection defaults to eager activation; startLanguageModel:false preserves the selected tuple and registered adapters without preloading through later readiness callbacks. Caller cancellation waits for accepted old-role cleanup and prevents later configuration/preload, or reaches the first owned load after selection commits. Newer accepted selections supersede older continuations; deferred or cancelled activation stays inactive until explicit load/startup, and a caller abort after successful settlement never unloads the ready model. Separate callers joining an existing runtime load retain their current cancellation ownership.
Import and lifecycle
import * as module from '/node_modules/arcane-os/runtime/arcane/modules/AI.js';
Browser import listens for user-entity-loaded and arcane-ollama-ready, installs window.ai once window.user is ready, emits ai-ready, and may perform provider configuration/transition, HTTP/native AI, microphone, and audio-playback side effects. User events prompt a readiness recheck without requiring their source detail to preserve window.user object identity; an immediate post-registration recheck catches an already-ready user without polling.
Application-facing behavior: Explicit null LLM service and null model constructor arguments leave LLM routes unselected; omitted/empty arguments retain defaults, and an existing window.ai is reused. supportsTranscriptionCapture() and createTranscriptionCapture(options) expose selected-provider capture without starting it. Native capture forwards exact PCM, callbacks and external endpoint/stop/cancel joins; finals are recognition segments, never automatic chat. Streaming native TTS preparation is inert, play consumes ordered PCM and released joins native cleanup. Per-utterance languageCode is forwarded separately from text. voice/speed/languageCode persist with an unfinished streamTTS utterance through finishTTS; explicit language change queues the previous remainder under its prior selection. Native fetchTTS(payload,signal) and fetchSTT(audio,signal) forward caller cancellation separately from their complete Core speech payloads, normalize aborts to AbortError and suppress late results; actual engine interruption remains service-owned. prepareTTSPlayback(payload,signal) silently returns a real Blob or native-speech descriptor; native controls expose finished/released/state/error/pause/resume/stop, and actual release owns the playback lane. fetchTTS and durable prepareTTS report ARCANEAITTSAUDIOEXPORTUNAVAILABLE for browser-native voices without audio export; Magpie retains full-audio export. Request-local model overrides remain exact and never mutate the configured model. Browser-speech protocol/event/error/reason constants; default AI; read-only providerRuntime, browserSpeechConfiguration, and browserSpeechDescriptor; explicit providerRuntime.status(role,{execution:true}) snapshots; streamTTS(text,end,options={}) with optional voice, speed, languageCode, pauseAfterMs, and waitForPlayback; default speech-input cleanup with SDK-internal preparation metadata and explicit complete-call textFormat:'plain' consumed before engine dispatch; prepareTTS({parts,storage,identity,textFormat,signal,onState}) returning ordered segments, state, ready, getAudio(index), and cancel(); async readPreparedTTS({parts,textFormat,storage,identity,signal,onState}) returning a complete ready handle or null through the same semantic matcher, with no activation, unmute, synthesis, playback, storage writes, or partial repair; manifest, audio availability, and returned-handle reads use DBOPFS createTable:false so absent tables remain absent; supplied adapters own that read-existing behavior; missing matches/audio return null, malformed metadata and storage failures reject, and omitted storage sees only completed matching same-AI in-memory audio; hits replay ready state, misses emit nothing, and the lookup signal detaches on settlement while returned-handle cancellation stays independent; prepareAudioOutput() returning the persistent WebAudio handle {kind,state,contextState,error,subscribe,resume,dispose} without resume, unmute, model load, sources, sound, or gesture listeners; current-state replay and output-specific canonical AIAUDIOOUTPUTSTATEEVENT (ai-audio-output-state), with ready only for an actually running context; explicit owner disposal observes context-bound cleanup and close; playPreparedTTS(prepared,{signal,onState,audioOutput}) returning state, error, finished, pause(), resume(), and stop() with one playback lane per AI and observational state/error callbacks; explicit live same-AI audioOutput borrows its context until owner disposal, while omission or null preserves per-playback terminal close; configure/dispose speech, route lifecycle, declaration-validated chat/stream requests with exact ordered calls; optional streamRequest toolText:{name,field} and onToolText(text,call,displayId) for decoded selected argument text with actual call identity, distinct from ordinary prose and terminal tool execution; synthesis/transcription and playback controls; initializes from current canonical user readiness, installs window.ai, and projects ai-ready plus active-generation ai-tts-failure. setAI(llmService,sttService,ttsService,model,modelTTS,modelSTT,options) and transitionAI(llmService,sttService,ttsService,model,modelTTS,modelSTT,options) preserve six positional preferences; configureProviders(selections,options) and transitionProviders(selections,options) accept the same optional {signal:null,startLanguageModel:true} defaults. setAI retains its synchronous boolean and configureProviders its synchronous route result. Explicit providerRuntime.load('llm',{signal}) or startProviders({startLanguageModel:true,signal}) activates a deferred selected route.
Protocol and host implementation
arcane-ai-browser-speech-configuration/1, AIProviderRuntime arcane-ai-provider/2 routes, globalThis.arcaneEvents, TWiN Cloud HTTPS, Arcane.ollama, Arcane.speech, Android WebView bridge This detail does not widen the application-facing API or grant authority.
Exports, signatures, parameters, and results
| Binding | Form | Declaration or signature | Parameter syntax |
|---|---|---|---|
AI_READY_EVENT | variable · value | variable AI_READY_EVENT | — |
AI_AUDIO_OUTPUT_STATE_EVENT | variable · value | variable AI_AUDIO_OUTPUT_STATE_EVENT | — |
AI_INITIALIZATION_ERROR_CODES | variable · value | variable AI_INITIALIZATION_ERROR_CODES | — |
AI_INITIALIZATION_REASONS | variable · value | variable AI_INITIALIZATION_REASONS | — |
AI_BROWSER_SPEECH_CONFIGURATION_PROTOCOL | variable · value | variable AI_BROWSER_SPEECH_CONFIGURATION_PROTOCOL | — |
AI_BROWSER_SPEECH_EVENT_TYPES | variable · value | variable AI_BROWSER_SPEECH_EVENT_TYPES | — |
AI_BROWSER_SPEECH_ERROR_CODES | variable · value | variable AI_BROWSER_SPEECH_ERROR_CODES | — |
AI_BROWSER_SPEECH_REASONS | variable · value | variable AI_BROWSER_SPEECH_REASONS | — |
default | default · class | class AI | — |
| Callable | Kind | Exact public signature | Parameter syntax |
|---|---|---|---|
AI.setAI | public-member | setAI(
llmService,
sttService,
ttsService,
model,
modelTTS,
modelSTT,
options
) |
llmService,
sttService,
ttsService,
model,
modelTTS,
modelSTT,
options
|
AI.configureProviders | public-member | configureProviders(selections,options) | selections,options |
AI.transitionAI | public-member | async transitionAI(
llmService,
sttService,
ttsService,
model,
modelTTS,
modelSTT,
options
) |
llmService,
sttService,
ttsService,
model,
modelTTS,
modelSTT,
options
|
AI.transitionProviders | public-member | async transitionProviders(selections,options) | selections,options |
AI.startProviders | public-member | async startProviders(options) | options |
AI.setSpeechMuted | public-member | async setSpeechMuted(muted) | muted |
AI.streamRequest | public-member | async streamRequest({
messages=[],
model,
structuredOutput=false,
localOnly=false,
speech=true,
onChunk=function ignoreStreamChunk(){},
toolText,
onToolText,
onComplete=function finishIgnoredStream(){},
onDataChunk=function ignoreStreamDataChunk(){},
onDataResult=function ignoreStreamDataResult(){},
onResponse=function ignoreStreamResponse(){},
tools=[],
toolChoice='auto',
onToolCall=function ignoreEarlyFunction(){},
onRequest=function ignoreStreamRequest(){},
onRetry = null,
parallelToolCalls,
id=Date.now(),
seeThinking=false,
signal=null,
maxOutputTokens,
maxTokens,
temperature,
topK,
topP,
repeatPenalty,
minP,
seed,
stop,
templateOptions,
reasoningEffort
}={}) | {
messages=[],
model,
structuredOutput=false,
localOnly=false,
speech=true,
onChunk=function ignoreStreamChunk(){},
toolText,
onToolText,
onComplete=function finishIgnoredStream(){},
onDataChunk=function ignoreStreamDataChunk(){},
onDataResult=function ignoreStreamDataResult(){},
onResponse=function ignoreStreamResponse(){},
tools=[],
toolChoice='auto',
onToolCall=function ignoreEarlyFunction(){},
onRequest=function ignoreStreamRequest(){},
onRetry = null,
parallelToolCalls,
id=Date.now(),
seeThinking=false,
signal=null,
maxOutputTokens,
maxTokens,
temperature,
topK,
topP,
repeatPenalty,
minP,
seed,
stop,
templateOptions,
reasoningEffort
}={} |
AI.streamMessage | public-member | async streamMessage(
messages=[],
streamHandler=function ignoreStreamChunk(){},
streamComplete=function finishIgnoredStream(){},
tools=[],
tool_choice='auto',
earlyFunctionTrigger=function ignoreEarlyFunction(){},
parallel_tool_calls,
id=Date.now(),
seeThinking=false,
signal=null,
requestHandler=function ignoreStreamRequest(){},
structuredOutput=false,
returnCompletion=false,
onDataChunk=function ignoreStreamDataChunk(){},
onDataResult=function ignoreStreamDataResult(){}
) |
messages=[],
streamHandler=function ignoreStreamChunk(){},
streamComplete=function finishIgnoredStream(){},
tools=[],
tool_choice='auto',
earlyFunctionTrigger=function ignoreEarlyFunction(){},
parallel_tool_calls,
id=Date.now(),
seeThinking=false,
signal=null,
requestHandler=function ignoreStreamRequest(){},
structuredOutput=false,
returnCompletion=false,
onDataChunk=function ignoreStreamDataChunk(){},
onDataResult=function ignoreStreamDataResult(){}
|
AI.fetchRequest | public-member | async fetchRequest({
messages=[],
model,
structuredOutput=false,
localOnly=false,
tools=[],
toolChoice='auto',
parallelToolCalls,
id=Date.now(),
signal=null,
onRequest=function ignoreFetchRequest(){},
onResponse=function ignoreFetchResponse(){},
onRetry = null,
maxOutputTokens,
maxTokens,
temperature,
topK,
topP,
repeatPenalty,
minP,
seed,
stop,
templateOptions,
reasoningEffort
}={}) | {
messages=[],
model,
structuredOutput=false,
localOnly=false,
tools=[],
toolChoice='auto',
parallelToolCalls,
id=Date.now(),
signal=null,
onRequest=function ignoreFetchRequest(){},
onResponse=function ignoreFetchResponse(){},
onRetry = null,
maxOutputTokens,
maxTokens,
temperature,
topK,
topP,
repeatPenalty,
minP,
seed,
stop,
templateOptions,
reasoningEffort
}={} |
AI.fetch | public-member | async fetch(
messages=[],
responseHandler=function ignoreFetchResponse(){},
structuredOutput=false,
tools=[],
tool_choice='auto',
parallel_tool_calls,
id=Date.now(),
requestHandler=function ignoreFetchRequest(){},
signal=null
) |
messages=[],
responseHandler=function ignoreFetchResponse(){},
structuredOutput=false,
tools=[],
tool_choice='auto',
parallel_tool_calls,
id=Date.now(),
requestHandler=function ignoreFetchRequest(){},
signal=null
|
AI.configureTTSSegmentation | public-member | configureTTSSegmentation(options={}) | options={} |
AI.streamTTS | public-member | streamTTS(
text='',
end=false,
options={}
) |
text='',
end=false,
options={}
|
AI.prepareTTS | public-member | prepareTTS({parts, textFormat, storage = null, identity = null, signal = null, onState = null} = {}) | {parts, textFormat, storage = null, identity = null, signal = null, onState = null} = {} |
AI.prepareTTSPlayback | public-member | async prepareTTSPlayback(payload={},signal=null,preparation={}) | payload={},signal=null,preparation={} |
AI.playPreparedTTS | public-member | playPreparedTTS(prepared, {signal = null, onState = null, audioOutput = null} = {}) | prepared, {signal = null, onState = null, audioOutput = null} = {} |
AI.finishTTS | public-member | finishTTS() | — |
AI.fetchTTS | public-member | async fetchTTS(payload={},signal=null,{speechInputPrepared=false}={}) | payload={},signal=null,{speechInputPrepared=false}={} |
AI.fetchSTT | public-member | async fetchSTT(audioFile,signal=null) | audioFile,signal=null |
AI.supportsTranscriptionCapture | public-member | supportsTranscriptionCapture() | — |
AI.createTranscriptionCapture | public-member | createTranscriptionCapture(options={}) | options={} |
AI.stopAudio | public-member | stopAudio() | — |
AI.resumeAudio | public-member | async resumeAudio(audioContext=null,fromUserGesture=true) | audioContext=null,fromUserGesture=true |
AI.playAudio | public-member | async playAudio(
audioChunks=[],
audioContext=null,
sourceNode=null,
audioType=this.audioType,
speechJob=null
) |
audioChunks=[],
audioContext=null,
sourceNode=null,
audioType=this.audioType,
speechJob=null
|
| Member | Kind | Exact public declaration | Parameter syntax |
|---|---|---|---|
AI.constructor | constructor | constructor(
llmService='',
sttService='',
ttsService='',
model='',
modelTTS='',
modelSTT=''
) |
llmService='',
sttService='',
ttsService='',
model='',
modelTTS='',
modelSTT=''
|
AI.ready | field | ready=false; | — |
AI.muted | field | muted=true; | — |
AI.llmService | field | llmService = ''; | — |
AI.sttService | field | sttService = ''; | — |
AI.ttsService | field | ttsService = ''; | — |
AI.model | field | model = ''; | — |
AI.modelTTS | field | modelTTS = ''; | — |
AI.modelSTT | field | modelSTT = ''; | — |
AI.reasoningEffort | field | reasoningEffort = ''; | — |
AI.audioFormat | field | audioFormat = 'opus'; | — |
AI.audioType | field | audioType = 'audio/ogg; codecs=opus'; | — |
AI.voiceSpeed | field | voiceSpeed = 1.0; | — |
AI.audioMessageChunks | field | audioMessageChunks=''; | — |
AI.sourceNodes | field | sourceNodes=[]; | — |
AI.isSpeaking | field | isSpeaking=false; | — |
AI.audioContext | field | audioContext=null; | — |
AI.currentSpeechJob | field | currentSpeechJob=null; | — |
AI.speechGeneration | field | speechGeneration=0; | — |
AI.speechJobs | field | speechJobs=[]; | — |
AI.speechAwaitingGesture | field | speechAwaitingGesture=false; | — |
AI.speechPlaybackStarting | field | speechPlaybackStarting=false; | — |
AI.speechResumeAttempt | field | speechResumeAttempt=0; | — |
AI.speechResumePending | field | speechResumePending=false; | — |
AI.speechScheduleGeneration | field | speechScheduleGeneration=0; | — |
AI.speechScheduleContext | field | speechScheduleContext=null; | — |
AI.speechScheduleTime | field | speechScheduleTime=0; | — |
AI.speechUnlockHandler | field | speechUnlockHandler=null; | — |
AI.providerRuntime | get | get providerRuntime() | — |
AI.speechActivationPending | get | get speechActivationPending() | — |
AI.AI_PUBLISH_READY | method | [AI_PUBLISH_READY]() | — |
AI.browserSpeechConfiguration | get | get browserSpeechConfiguration() | — |
AI.browserSpeechDescriptor | get | get browserSpeechDescriptor() | — |
AI.url | get | get url() | — |
AI.urlTTS | get | get urlTTS() | — |
AI.urlSTT | get | get urlSTT() | — |
AI.twinKey | get | get twinKey() | — |
AI.twinKey | set | set twinKey(value) | value |
AI.license | get | get license() | — |
AI.license | set | set license(value) | value |
AI.configured | get | get configured() | — |
AI.setAI | method | setAI(
llmService,
sttService,
ttsService,
model,
modelTTS,
modelSTT,
options
) |
llmService,
sttService,
ttsService,
model,
modelTTS,
modelSTT,
options
|
AI.configureProviders | method | configureProviders(selections,options) | selections,options |
AI.configureSpeechProviders | method | configureSpeechProviders(selections) | selections |
AI.transitionAI | async method | async transitionAI(
llmService,
sttService,
ttsService,
model,
modelTTS,
modelSTT,
options
) |
llmService,
sttService,
ttsService,
model,
modelTTS,
modelSTT,
options
|
AI.transitionProviders | async method | async transitionProviders(selections,options) | selections,options |
AI.transitionSpeechProviders | async method | async transitionSpeechProviders(selections) | selections |
AI.configureSpeechProvider | method | configureSpeechProvider(role,provider,options={}) | role,provider,options={} |
AI.configureBrowserSpeech | method | configureBrowserSpeech(configuration,options={}) | configuration,options={} |
AI.disposeBrowserSpeech | method | disposeBrowserSpeech(options={}) | options={} |
AI.startProviders | async method | async startProviders(options) | options |
AI.setSpeechMuted | async method | async setSpeechMuted(muted) | muted |
AI.streamRequest | async method | async streamRequest({
messages=[],
model,
structuredOutput=false,
localOnly=false,
speech=true,
onChunk=function ignoreStreamChunk(){},
toolText,
onToolText,
onComplete=function finishIgnoredStream(){},
onDataChunk=function ignoreStreamDataChunk(){},
onDataResult=function ignoreStreamDataResult(){},
onResponse=function ignoreStreamResponse(){},
tools=[],
toolChoice='auto',
onToolCall=function ignoreEarlyFunction(){},
onRequest=function ignoreStreamRequest(){},
onRetry = null,
parallelToolCalls,
id=Date.now(),
seeThinking=false,
signal=null,
maxOutputTokens,
maxTokens,
temperature,
topK,
topP,
repeatPenalty,
minP,
seed,
stop,
templateOptions,
reasoningEffort
}={}) | {
messages=[],
model,
structuredOutput=false,
localOnly=false,
speech=true,
onChunk=function ignoreStreamChunk(){},
toolText,
onToolText,
onComplete=function finishIgnoredStream(){},
onDataChunk=function ignoreStreamDataChunk(){},
onDataResult=function ignoreStreamDataResult(){},
onResponse=function ignoreStreamResponse(){},
tools=[],
toolChoice='auto',
onToolCall=function ignoreEarlyFunction(){},
onRequest=function ignoreStreamRequest(){},
onRetry = null,
parallelToolCalls,
id=Date.now(),
seeThinking=false,
signal=null,
maxOutputTokens,
maxTokens,
temperature,
topK,
topP,
repeatPenalty,
minP,
seed,
stop,
templateOptions,
reasoningEffort
}={} |
AI.streamMessage | async method | async streamMessage(
messages=[],
streamHandler=function ignoreStreamChunk(){},
streamComplete=function finishIgnoredStream(){},
tools=[],
tool_choice='auto',
earlyFunctionTrigger=function ignoreEarlyFunction(){},
parallel_tool_calls,
id=Date.now(),
seeThinking=false,
signal=null,
requestHandler=function ignoreStreamRequest(){},
structuredOutput=false,
returnCompletion=false,
onDataChunk=function ignoreStreamDataChunk(){},
onDataResult=function ignoreStreamDataResult(){}
) |
messages=[],
streamHandler=function ignoreStreamChunk(){},
streamComplete=function finishIgnoredStream(){},
tools=[],
tool_choice='auto',
earlyFunctionTrigger=function ignoreEarlyFunction(){},
parallel_tool_calls,
id=Date.now(),
seeThinking=false,
signal=null,
requestHandler=function ignoreStreamRequest(){},
structuredOutput=false,
returnCompletion=false,
onDataChunk=function ignoreStreamDataChunk(){},
onDataResult=function ignoreStreamDataResult(){}
|
AI.fetchRequest | async method | async fetchRequest({
messages=[],
model,
structuredOutput=false,
localOnly=false,
tools=[],
toolChoice='auto',
parallelToolCalls,
id=Date.now(),
signal=null,
onRequest=function ignoreFetchRequest(){},
onResponse=function ignoreFetchResponse(){},
onRetry = null,
maxOutputTokens,
maxTokens,
temperature,
topK,
topP,
repeatPenalty,
minP,
seed,
stop,
templateOptions,
reasoningEffort
}={}) | {
messages=[],
model,
structuredOutput=false,
localOnly=false,
tools=[],
toolChoice='auto',
parallelToolCalls,
id=Date.now(),
signal=null,
onRequest=function ignoreFetchRequest(){},
onResponse=function ignoreFetchResponse(){},
onRetry = null,
maxOutputTokens,
maxTokens,
temperature,
topK,
topP,
repeatPenalty,
minP,
seed,
stop,
templateOptions,
reasoningEffort
}={} |
AI.fetch | async method | async fetch(
messages=[],
responseHandler=function ignoreFetchResponse(){},
structuredOutput=false,
tools=[],
tool_choice='auto',
parallel_tool_calls,
id=Date.now(),
requestHandler=function ignoreFetchRequest(){},
signal=null
) |
messages=[],
responseHandler=function ignoreFetchResponse(){},
structuredOutput=false,
tools=[],
tool_choice='auto',
parallel_tool_calls,
id=Date.now(),
requestHandler=function ignoreFetchRequest(){},
signal=null
|
AI.ttsSegmentation | get | get ttsSegmentation() | — |
AI.configureTTSSegmentation | method | configureTTSSegmentation(options={}) | options={} |
AI.readPreparedTTS | async method | async readPreparedTTS({parts, textFormat, storage = null, identity = null, signal = null, onState = null} = {}) | {parts, textFormat, storage = null, identity = null, signal = null, onState = null} = {} |
AI.prepareTTS | method | prepareTTS({parts, textFormat, storage = null, identity = null, signal = null, onState = null} = {}) | {parts, textFormat, storage = null, identity = null, signal = null, onState = null} = {} |
AI.playPreparedTTS | method | playPreparedTTS(prepared, {signal = null, onState = null, audioOutput = null} = {}) | prepared, {signal = null, onState = null, audioOutput = null} = {} |
AI.streamTTS | method | streamTTS(
text='',
end=false,
options={}
) |
text='',
end=false,
options={}
|
AI.finishTTS | method | finishTTS() | — |
AI.prepareAudioOutput | method | prepareAudioOutput() | — |
AI.prepareTTSPlayback | async method | async prepareTTSPlayback(payload={},signal=null,preparation={}) | payload={},signal=null,preparation={} |
AI.fetchTTS | async method | async fetchTTS(payload={},signal=null,{speechInputPrepared=false}={}) | payload={},signal=null,{speechInputPrepared=false}={} |
AI.supportsTranscriptionCapture | method | supportsTranscriptionCapture() | — |
AI.createTranscriptionCapture | method | createTranscriptionCapture(options={}) | options={} |
AI.fetchSTT | async method | async fetchSTT(audioFile,signal=null) | audioFile,signal=null |
AI.stopAudio | method | stopAudio() | — |
AI.resumeAudio | async method | async resumeAudio(audioContext=null,fromUserGesture=true) | audioContext=null,fromUserGesture=true |
AI.playAudio | async method | async playAudio(
audioChunks=[],
audioContext=null,
sourceNode=null,
audioType=this.audioType,
speechJob=null
) |
audioChunks=[],
audioContext=null,
sourceNode=null,
audioType=this.audioType,
speechJob=null
|
AI.nextSentance | method | nextSentance(job=this.currentSpeechJob) | job=this.currentSpeechJob |
Parameter meanings and results
AI(llmService, sttService, ttsService, model, modelTTS, modelSTT) selects providers and models. Explicit null in both llmService and model leaves the LLM unselected from construction; omitted/empty values preserve defaults and an existing window.ai remains the singleton. setAI(llmService, sttService, ttsService, model, modelTTS, modelSTT, options) and configureProviders(config, options) preserve synchronous boolean/route returns. transitionAI(llmService, sttService, ttsService, model, modelTTS, modelSTT, options) and transitionProviders(selections, options) stop queued audio, concurrently unload the three current roles, and then configure replacements; the first returns aggregate runtime status and the second returns the configured routes. All four selection APIs accept optional {signal=null,startLanguageModel=true}. The default remains eager. Explicit false registers built-in adapters and updates the AI tuple and runtime routes without preloading, including through later readiness/credential/residency callbacks. Explicit providerRuntime.load or startProviders can activate the deferred selection. An already-aborted signal changes no selection. Accepted old cleanup settles before cancellation prevents replacement configuration/preload; once committed, cancellation reaches the first owned load and preserves the selected route with its actual state. Newer accepted selections supersede older pending transitions; stale completion cannot configure, load, or clean up the newer selection. Cancelled activation is not restarted by readiness callbacks. Completed settlement detaches its caller signal, so later abort does not unload the ready model. Same-route runtime-load joining retains its existing cancellation ownership. startProviders({startLanguageModel=true,startMuted=true,startTranscription=false,signal=null}={}) declines the startup STT load unless explicitly requested without forcing an independently active STT role to unload, and setSpeechMuted(muted) owns TTS lifecycle. configureTTSSegmentation({punctuation,wordCadence}) selects sentence, any-punctuation, or no-punctuation boundaries plus an optional positive whole-word cadence without changing or discarding text; synthesis enters provider-capacity FIFO admission immediately and ready audio buffers play consecutively in original order on their owning AudioContext clocks. streamTTS() and finishTTS() automatically remove repeated same formatting marks from only a cloned outbound speech-input copy; caller, displayed, stored, and model text remains exact. Complete fetchTTS() and audio-file prepareTTSPlayback() calls use that cleanup by default; explicit textFormat:'plain' preserves supplied speech input and is consumed before engine dispatch. Omitted or other values retain default cleanup, and the option changes neither streaming cleanup nor upstream engine capacity; SpeechPlayback independently accepts the same public plain option in its constructor or prepare call. The optional fetchTTS(payload,signal,preparation={}) preparation argument is SDK-internal, and {speechInputPrepared:true} prevents a second non-idempotent cleanup pass after an SDK owner has already prepared the copy; applications omit it. streamTTS(text,end,options={}) supports third-argument voice/speed options from SDK 0.5.12 and adds languageCode in SDK 0.95.0; it captures supplied voice, speed and languageCode per extracted segment without changing instance defaults; omission preserves an unfinished utterance selection or uses current defaults for a new utterance. pauseAfterMs defaults to 0 and places finite nonnegative milliseconds after only the final extracted segment on the existing audio clock; invalid pauses throw RangeError. waitForPlayback defaults to false and retains the preparation boolean. With true, the boolean settles after every segment extracted by that invocation reaches a terminal state: true when all naturally end, false on terminal cancellation or failure. Voice, speed and languageCode are retained with the unfinished utterance through continuation and finishTTS(); supplied values replace that selection. An explicit language change first queues the preceding remainder with its old selection. End, mute and stop clear it. Pause and waitForPlayback remain per-call. Without a preceding language-change flush, no extracted segment means immediate true without waiting for earlier jobs; a language-change call also joins its preceding remainder preparation; muted calls return false unless an explicit speech activation is pending. Autoplay and recoverable resume waiting remain pending; a failed resume of a closed AudioContext terminates affected jobs with false playback results. A trailing pause delays the next queued audio, not the completed buffer promise. finishTTS() remains a preparation flush. stopAudio() cancels AI-owned streamed speech and prepared playback and settles pending playback promises false, while detached preparation continues. prepareTTS({parts,storage,identity,textFormat,signal,onState}) returns {segments,state,ready,getAudio(index),cancel()} synchronously, retains exact source semantic inputs, applies default speech formatting cleanup once or preserves supplied input for explicit textFormat:'plain' before the existing configured punctuation segmentation, and prepares without playback through the existing bounded provider queue. Optional storage {db,table,key} persists complete audio and MIME metadata in caller-owned DBOPFS and reuses matching semantic variants. Matching pending requests share synthesis on the same AI and storage group; cancelling one observer only cancels shared generation when no active observer remains. ready resolves after complete generation/reuse and optional durable saving, or rejects the complete error or AbortError; completed stored segments survive. onState synchronously receives {state,completed,total,segments,error}. Async readPreparedTTS({parts,textFormat,storage,identity,signal,onState}) resolves a complete ready preparation handle or null using the same original parts, speech selection/configuration, voice/speed/pause values, text treatment, segmentation, and identity matcher as prepareTTS. It never activates or unmutes a model, synthesizes, plays, writes storage, or repairs partial audio. Storage lookup reads the existing manifest and every ordered audio file through the same per-key queue. Manifest, audio availability, and retained handle reads pass createTable:false to DBOPFS, leaving missing tables absent; supplied adapters own that read-existing behavior. Already-started non-abortable reads may settle after cancellation, while observed cancellation prevents further reads or returning a handle; no matching record or any missing audio returns null, while malformed metadata and other storage errors reject. Without storage it reads only completed matching same-AI retained in-memory preparations; pending or failed work returns null. A hit synchronously replays ready state through onState and returns all segments ready; a miss emits nothing. Lookup cancellation rejects with AbortError and the signal detaches on settlement, leaving the returned handle independent of later lookup abort; handle cancel remains explicit. Durable getAudio rereads files with retained MIME metadata, so later missing files remain observable errors. Native speech without audio export rejects ARCANE_AI_TTS_AUDIO_EXPORT_UNAVAILABLE as prepareTTS does. prepareAudioOutput() synchronously creates or reuses the persistent WebAudio context without resuming, unmuting, loading a model, creating sources, playing sound, or registering gesture listeners. Its same-owner handle exposes kind:web-audio, live state/contextState/error, subscribe(listener), resume(), and dispose(); repeated calls return that handle until disposed or closed. subscribe immediately replays {kind,state,contextState,error} and observes only this output through AI_AUDIO_OUTPUT_STATE_EVENT (ai-audio-output-state), whose canonical occurrence includes detail and operationId; unsubscribe is idempotent. ready means the actual context is running, not HTML audio, native speech, model readiness, future autoplay permission, or audible delivery. Other states retain platform context state, gesture-required after observed NotAllowedError, unavailable, error, disposing, disposed, or closed. Explicit resume resolves a boolean for actual readiness and rejects actual platform errors while retaining them in state. dispose stops context-bound playback and dependent clock work, observes cleanup, awaits actual close, releases subscriptions, and preserves detached preparation, saved audio, and unrelated contexts. Concurrent disposal calls share the promise; actual failures remain observable and failed close is retryable. playPreparedTTS(prepared,{signal,onState,audioOutput}) returns {state,error,finished,pause(),resume(),stop()} immediately, reuses the ordered audio-clock scheduler, and enables cached playback without loading the selected model; missing generation alone requests provider readiness. State/error getters and optional observational onState({state,error}) expose waiting, waiting-for-gesture, scheduled, paused, complete, stopped, or error; waiting-for-gesture identifies audio-unlock UI, while a false resume result alone is not a first-segment-ready signal. One AI has one playback lane: attaching prepared playback replaces preceding streamed/prepared audio, and streamTTS interrupts active prepared playback. finished is true after all audio naturally ends or false after stop, cancellation, or failure; final trailing silence does not delay completion, and failures use existing diagnostics/events. Omitted or null audioOutput preserves terminal closure of the playback-owned context; an explicit live same-AI output handle lends its exact context and remains open after playback completion or stop until owner disposal. Another owner, closed, or disposing handle throws TypeError. Playback controls and replacement do not cancel preparation, pause/resume affect only that playback audio context, and mute or provider replacement retain their broader provider lifecycle authority. Preparation cancellation prevents later synthesis and releases its pending TTS activation interest. The final pending interest aborts the owned activation; other preparation, explicit playback activation, or joining direct TTS-load interests preserve it. Complete cache hits never activate the model, and later cancellation does not unload an already-ready model. Provider work owns actual cancellation and cleanup completion. fetchSTT(audioFile,signal=null) returns the transcript and propagates the supplied cancellation signal. Selected TWiN Cloud TWIN LLM and Core LOCAL_SPEACH STT/TTS routes expose capability readiness through provider/2 adapters. The OLLAMA LLM adapter explicitly preloads the selected model with empty generate and confirms actual residency through running() before reporting ready. Speech preference slots remain selected as supplied, including pending unregistered external routes. STT wrapper configurations remain local-only; explicit TTS may be remote. configureSpeechProvider(role,provider,{modelId,signal,expectedProvider}) serializes one-role registration/replacement/removal on the existing speech configuration lane, preserves the other role and LLM, disposes unaccepted candidates, and reports committed failures. modelId defaults to the first catalog entry. Passing null removes that role; a mismatched expectedProvider resolves false without changing another owner. Browser-local configuration and explicit remote TTS compose in either order without configuration-time network work. setSpeechMuted(false) may wait for the selected provider to register or finish configuration; speechActivationPending exposes that explicit intent. streamTTS and finishTTS preserve arriving text in the existing queue during that wait without initiating activation themselves. Initial same-provider/model registration preserves queued speech. Stop cancels queued preparation promptly; mute, disposal, a new startup operation, or actual route replacement revoke pending activation. The TWiN access key is used only for remote LLM chat. Before importing AI.js or any module that imports it, hydrating a ready window.user, applying saved preferences, or starting providers, only exact uppercase OPENAI in tuple slot 0 (LLM provider) or slot 3 (default-model sentinel) changes to TWIN; all other values, openai-gpt-oss-120b, openai-gpt-oss-20b, OpenAI-compatible wire terminology, and separate Core provider:openai behavior remain unchanged; the SDK does not alias that identifier or rewrite saved preferences. TTS fills an omitted voice from the exact selected model catalog defaultVoice; a saved OpenAI-route voice is not forwarded to Core or browser speech. Every active-generation non-abort TTS synthesis, decode, playback-start, or playback-resume failure emits ai-tts-failure with its complete Error and exact operation boundary; mute, cancellation, permission waiting, and superseded generations emit no failure and provider readiness is not rewritten. fetchRequest(options) preserves provider-native JSON. Both object-form fetchRequest and streamRequest accept an optional request-local model: TWiN HTTP and native Ollama forward its exact value, omission or undefined preserves the configured model, and concurrent requests never mutate ai.model or provider selection. Built-in request observers and retries retain that model; stream/native completion metadata uses it only when the provider omits model. Registered adapters receive model only when supplied and own its interpretation; this does not switch or activate a loaded browser-WASM model. Both object-form methods forward optional caller-owned temperature unchanged to built-in TWiN Cloud, including its provider-runtime adapter; explicit 0 is preserved, omission or undefined leaves the field absent, retries reuse the value, and concurrent requests remain independent. There is no SDK clamp, saved preference, output limit, or native Ollama option mapping. Other registered providers retain their existing interpretation. fetchRequest(options) and streamRequest(options) accept provider-neutral reasoningEffort values none, low, medium, high, and max; TWiN Cloud maps a supplied value to DigitalOcean reasoning_effort, omission preserves the provider default, and explicit openai-gpt-oss-120b or openai-gpt-oss-20b selection is preserved. streamRequest(options) forwards provider chunks after private structural fields are removed through onDataChunk, every choice's nonstructural content/reasoning through onChunk in provider order, the complete terminal response through onDataResult and onResponse, each complete validated structural call exactly once through onToolCall, and then awaits onComplete. Optional toolText:{name,field} with onToolText(text,call,displayId) emits newly decoded text from one selected root string argument in arrival order, preserving whitespace and characters decoded from JSON escapes. call is {id,name,field,index,choiceIndex} with the actual normalized call ID; delivery waits for the matching name and ID. displayId is the same M-request display ID used by onChunk. This distinct callback does not add ordinary prose, alter arguments, execute tools, or persist history. Complete-only providers emit text only when their actual complete response arrives, and repeated terminal snapshots do not replay emitted text. Omitting toolText preserves existing behavior; callback failures reach the request owner and cancellation prevents later delivery. Its application result is the ordered structural-call array for selected tool output, the complete completion object for multiple choices, or the ordinary single-result text/completion otherwise; partial structural deltas remain private. onRequest receives supplied transport metadata. Native Ollama calls are adapted before validation without inventing arguments.message. Browser speech normalizes shared Blob/File STT and selected-provider audio requests at the provider boundary. Whisper STT and Kokoro TTS default execution.device to auto: request webnn-npu when navigator.ml.createContext is exposed, then webgpu when navigator.gpu is exposed, then wasm for CPU execution. Failed backend loads clean up the pool before trying fresh Workers with the same prepared model and dtype; explicit webnn-npu, webgpu, or wasm never falls back. Whisper maxConcurrentRequests defaults to and only accepts 1. Kokoro defaults to four Worker/model sessions; explicit execution capacity 1 through 4 remains supported. Fifth and later segments wait in the FIFO queue without dropping content, while playback schedules exact input order. Raising capacity trades memory for latency. Explicit providerRuntime.status(null,{execution:true}) exposes roles.stt.execution and roles.tts.execution with provider-supplied requestedDevice, selectedDevice, maxConcurrentRequests, and activeRequestCount; selectedDevice is null while unloaded. The selectedDevice names the successful upstream session backend request, not proof that every operation used the physical NPU; WebNN unsupported operations may use WASM, and compatibility depends on the browser, driver, hardware, and graph. Default status preserves sticky-state identity and never probes the provider. Execution reports do not establish physical GPU kernel overlap. Browser-native TTS catalogs declare speech.playback native. prepareTTSPlayback(payload,signal) returns real Blob audio or an inert native-speech descriptor whose play options return finished/released/state/error/pause/resume/stop. Native preparation preserves complete input and selected catalog or per-request voice/language/speed without global user preference injection. Original cancellation remains attached by default to later play. Native stream jobs use actual completion and retain failed controls through actual resource release; buffered playback cannot overlap unreleased native speech. Explicit saved-audio contexts still resume normally. fetchTTS and durable prepareTTS report ARCANE_AI_TTS_AUDIO_EXPORT_UNAVAILABLE for browser-native synthesis without audio export rather than fabricating a file. For a selected native streaming engine such as Magpie, prepareTTSPlayback is inert until play consumes ordered PCM through the existing provider lane and shared audio output; full fetchTTS remains available. Per-utterance languageCode stays separate from input; no language classifier is introduced. supportsTranscriptionCapture() reports the selected capture capability; createTranscriptionCapture(options) returns null or an owned start/stop/cancel/destroy/done handle with optional forceEndpoint. Native capture keeps actual sample rate, exact PCM and complete snapshots. Callbacks receive (result,{requestStop}); request a drain without awaiting it, then external code joins stop/done. External forceEndpoint flushes pending audio and recognition without closing microphone or stream scope. ASR final segments do not decide conversation completion or send chat.
Events, side effects, and errors
Source-literal CustomEvent dispatches
No source-literal CustomEvent dispatch is part of this artifact.
Lifecycle and event flow
- consumes user-entity-loaded
- consumes arcane-ollama-ready
- emits ai-ready
- emits ai-tts-failure
- emits ai-audio-output-state
Direct coded failures
AI_REASONING_EFFORT_INVALIDAI_NATIVE_LOCAL_REQUIREDAI_NATIVE_LOCAL_REQUIREDAI_PROVIDER_NOT_CONFIGUREDAI_MODEL_INVALIDAI_STT_DEVICE_ONLYARCANE_AI_PROVIDER_UNAVAILABLEAI_STT_DEVICE_ONLYARCANE_AI_TTS_NATIVE_AUDIO_INVALIDAI_ANDROID_NATIVE_SPEECH_UNAVAILABLEAI_REQUIRED_TOOL_UNAVAILABLEAI_TOOL_ARGUMENTS_INVALIDAI_TOOL_ARGUMENTS_INVALIDAI_REQUIRED_TOOL_CALL_MISSINGAI_LOCAL_MODEL_REQUIREDAI_LOCAL_MODEL_REQUIREDARCANE_AI_TTS_AUDIO_EXPORT_UNAVAILABLEARCANE_AI_TTS_PROVIDER_AUDIO_INVALIDARCANE_AI_TTS_AUDIO_EXPORT_UNAVAILABLEARCANE_AI_TTS_REQUEST_INVALIDARCANE_AI_TTS_REQUEST_INVALIDARCANE_AI_TTS_SIGNAL_INVALIDARCANE_AI_TTS_INPUT_INVALIDARCANE_AI_TTS_MODEL_INVALIDARCANE_AI_TTS_MODEL_REQUIREDARCANE_AI_TTS_MODEL_SELECTION_MISMATCHARCANE_AI_TTS_VOICE_INVALIDARCANE_AI_TTS_VOICE_REQUIREDARCANE_AI_TTS_RESPONSE_FORMAT_INVALIDARCANE_AI_TTS_RESPONSE_FORMAT_INVALIDARCANE_AI_TTS_SPEED_INVALIDARCANE_AI_STT_SIGNAL_INVALIDARCANE_AI_STT_PROVIDER_TRANSCRIPT_INVALID
Exported Error subclasses
This artifact exports no Error subclass.
Documented failure behavior
- AI_NATIVE_LOCAL_REQUIRED
- AI_PROVIDER_NOT_CONFIGURED
- AI_MODEL_INVALID
- AI_LOCAL_MODEL_REQUIRED
- AI_REASONING_EFFORT_INVALID
- ARCANE_AI_TOOL_TEXT_INVALID
- AI_STRUCTURED_OUTPUT_INVALID
- AI_REQUIRED_TOOL_UNAVAILABLE
- AI_REQUIRED_TOOL_CALL_MISSING
- AI_SERVICE_UNREACHABLE
- AI_REQUEST_FAILED
- AI_REQUEST_ABORTED
- AI_ANDROID_NATIVE_SPEECH_UNAVAILABLE
- AI_STT_DEVICE_ONLY
- ARCANE_AI_MODEL_AUTHORITY_REQUIRED
- ARCANE_AI_PROVIDER_DISPOSED
- ARCANE_AI_PROVIDER_RUNTIME_INVALID
- ARCANE_AI_PROVIDER_UNAVAILABLE
- ARCANE_AI_REQUEST_ABORTED
- ARCANE_AI_OPERATION_SUPERSEDED
- ARCANE_AI_ROLE_BUSY
- ARCANE_AI_ROLE_NOT_READY
- ARCANE_AI_INVALID_REQUEST
- ARCANE_AI_AUDIO_DECODE_UNAVAILABLE
- ARCANE_AI_AUDIO_DECODE_FAILED
- ARCANE_AI_INVALID_PROVIDER_RESULT
- ARCANE_AI_UNSUPPORTED_RESPONSE_FORMAT
- ARCANE_AI_TTS_HTTP_REQUEST_FAILED
Availability and capabilities
Browser + native bridge + TWiN Cloud. Complete mutable caller-owned browser speech authority, SDK-owned provider registration/replacement/disposal, explicit STT/TTS activation, normalized Whisper STT and Kokoro TTS execution with default capacities 1 and 4 respectively, explicit provider-neutral execution snapshots through status(role,{execution:true}), sticky readiness, canonical window.user readiness without compatibility-event object-identity admission or polling, exact ordered structural calls with required arguments.message, atomic nonblank all-ID tool-result sequencing, complete all-choice streaming/data validation, native Ollama structural adaptation, complete-response then all validated tool callbacks then completion ordering, observed async callbacks, Blob/File transcription, default repeated-formatting-mark removal from cloned outbound TTS input, explicit textFormat:'plain' preservation for complete synthesis/preparation while streaming cleanup remains automatic and caller/visible content stays exact, immediate exact-segment synthesis, per-call voice/speed capture, optional final-segment pause on the existing audio clock, optional per-call terminal playback completion, detached preparation with complete semantic DBOPFS audio reuse and same-owner pending request sharing; each preparation owns its pending activation interest, final-interest cancellation aborts that activation while other interests preserve it, complete cache hits start no model, and cancellation after readiness leaves the ready model loaded; independent prepared playback without a model load for cached audio, ordered audio-clock scheduling, playable audio, active-generation TTS operation-failure routing, mute, and cancellation are normalized; provider/model/runtime/voice policy remains caller-owned. Both speech roles default execution.device to auto: webnn-npu when navigator.ml.createContext is exposed, then webgpu when navigator.gpu is exposed, then wasm for CPU execution. Failed backend loads clean up the pool before trying fresh Workers with the same prepared model and dtype; explicit devices never fall back. STT capacity only accepts 1; TTS accepts 1 through 4. Execution snapshots expose requestedDevice, selectedDevice, maxConcurrentRequests, and activeRequestCount for both roles; selectedDevice is null while unloaded and names the successful upstream session backend request when loaded. It does not prove every operation used the physical NPU: WebNN unsupported operations may use WASM, and compatibility depends on the browser, driver, hardware, and graph. High-level LLM selection defaults to eager activation; startLanguageModel:false preserves the selected tuple and registered adapters without preloading through later readiness callbacks. Caller cancellation waits for accepted old-role cleanup and prevents later configuration/preload, or reaches the first owned load after selection commits. Newer accepted selections supersede older continuations; deferred or cancelled activation stays inactive until explicit load/startup, and a caller abort after successful settlement never unloads the ready model. Separate callers joining an existing runtime load retain their current cancellation ownership.
Normalizes explicitly selected provider/2 LLM, on-device Whisper STT, and on-device Kokoro or explicitly configured remote TTS routes alongside configured TWiN Cloud HTTPS, Arcane.ollama.chat, and Arcane.speech paths. Each route keeps its real browser, native/Core, or cloud availability and never silently falls back.
Contract example
import AI from '/node_modules/arcane-os/runtime/arcane/modules/AI.js';
// The application supplies this runtime argument; never put its key in source.
async function sayHello(applicationRuntime) {
const ai = new AI();
ai.twinKey = applicationRuntime.twinKey;
try {
const response = await ai.fetchRequest(
{
messages: [{role: 'user', content: 'Hello!'}]
}
);
console.log(JSON.stringify(response, null, 2));
} catch (error) {
console.error(error.code, error.message);
}
}