9i'UddlmZddlZddlmZmZddlmZddlm Z m Z m Z m Z m Z mZddlmZmZddlZddlZddlmZddlmZdd lmZe eefZiZd ed <ej<Zdd Z dd Z!ddZ"ddZ#ddZ$ddZ%dddZ&y)) annotationsN)SentenceTransformermodels)Path)IterableUnionListSetAnyDict) AutoTokenizerPreTrainedTokenizerBase) QdrantClient) config_param) CONFIG_DIRz"dict[str, PreTrainedTokenizerBase]_TOKENIZER_CACHEctd|}tt|dxsi}t|tr|j dini}t|ts t d|S)N/profiles) config_filefield eqp_manualszz attach_helpers..%s% B $(B ,.B r#encode_documentc,|j|fddi|Sr'r)r+s r!r/z attach_helpers..)s%KDKK E $(E ,.E r#N)hasattrrr%r0)sts r!attach_helpersr4s6 2~ &, ( 2( )/ + *r#cZ t||}t|||_ |S#ttf$rx}td|d|t j |t jddd}t j|jd}t||g|}Yd }~d }~wwxYw) a Loads a SentenceTransformer model instance from a local directory. If loading directly fails, a fallback mechanism is implemented to initialize the model using specific transformer and pooling components. This function also attaches helper methods to the SentenceTransformer instance and sets the maximum sequence length for the model. :param model_dir: The directory path where the model files are stored. :type model_dir: Str :param device: The computation device to be used (e.g., "cpu" or "cuda"). :type device: Str :param max_seq_len: The maximum sequence length to be set for the model. :type max_seq_len: Int :return: Loaded and configured SentenceTransformer model. :rtype: SentenceTransformer )devicez-Error loading SentenceTransformer model from z: T) torch_dtypelocal_files_onlylow_cpu_mem_usage) model_args)pooling_mode_mean_tokens)modulesr6N) rFileNotFoundErrorrprintr Transformertorchfloat32Poolingget_word_embedding_dimensionr4max_seq_length) model_dirr6 max_seq_lenr3ewordpools r! load_local_strJ-s" F 6 :2#B I z *F =i[1#NO!! ',}}$eij ~~d??A\`a $f EFs #B*A.B%%B*ct|ts2tjjt |vs dt |vr#t t|j }n t |}|tvr t|St5|tvrt|cdddS tj|dd}|t|<|cdddS#t$rtj|d}Y6wxYw#1swYyxYw)NrT)use_fastr8)rL) rrospathseprresolver _TOK_LOCKr from_pretrained Exception)rEkeytoks r! get_tokenizerrVMs)T"bggkkS^&CscR[nG\$y/))+,)n $$  " "#C(  D//dUYZC!$   D//dCC D s0C6$C< C6 C30C62C33C66C?cDt|}t||ddS)NF)add_special_tokens input_ids)rVlen)textrErUs r! count_tokensr\as#  "C s4E2;? @@r#ct|}|t|ddd}|dDcgc] }t|c}Scc}w)NF)rXpadding truncationrY)rVlistrZ)textsrErUencidss r!count_tokens_batchrdfsC  "C d5keUu UC #K 0 1 0CH 0 11 1s=ct}d} |j|d|dd|\}}|s t |S|D]@}|jxsi}|jd}|s'|j t |B|s t |S|)zTScroll the collection and return a sorted list of distinct payload['doc_id'] values.NTF)collection_name scroll_filterlimit with_payload with_vectorsoffsetdoc_id)setscrollpayloadraddrsorted) client collectionbatchdoc_ids next_pagepointsprods r! list_doc_idsrzlsGI "MM& *    '?A&'iio2G H%A CF#    '?% r#)rrreturnr)r3rr{None)rErr6rrFintr{r)rE ModelPathr{r)r[rrEr~r{r})raz Iterable[str]rEr~r{z List[int])i)rrrrsrrtr}r{z List[str])' __future__rr@sentence_transformersrrpathlibrtypingrrr r r r transformersr r threadingrM qdrant_clientraom.utils.utilitiesraom.definitionsrrr~r__annotations__RLockrQr"r4rJrVr\rdrzr#r!rsw# =88?&,& #t)  7949 IOO   @(A 2 r#