4j^jddlmZddlZddlZddlZddlmZmZmZddl m Z ddl m Z ddl mZddlZddlmZer ddlZddlZddlZiZ dZ ed eDZdd Zdd Z dd Z d dd ZGddZ d ddZy)) annotationsN)CallableSequence TYPE_CHECKING) lru_cache) accumulate)suppress)tqdm).?!*;,()[]u“u”u‘u’'"`:u—u…/\u–&-c#FK|]}tj|ywN)reescape).0splitters D/home/agent/.local/lib/python3.12/site-packages/semchunk/semchunk.py r$:s 9AQY89L9As!c d}d|vsd|vr'ttjd|t}nd|vr&ttjd|t}ntjd|rttjd |t}t|d k(rt D]d}tj|d |x}s|j d }tj|}||tjd |d ||fcSn!tD] }||vsd}nd|t|fS|||j|fS)zDSplit text using the most semantically meaningful splitter possible.T  z[\r\n]+)key z\t+z\sz\s+z(\s)z(?<=rF) maxrfindalllensearch0_REGEX_ESCAPED_NON_WHITESPACE_SEMANTIC_SPLITTERSgroupr split"_NON_WHITESPACE_SEMANTIC_SPLITTERSlist)textsplitter_is_whitespacer"escaped_precederwhitespace_preceded_by_precederescaped_splitters r# _split_textr:<sO" t|tt|rzz*d3= rzz&$/S9 5$ rzz&$/S9 x=A $T { 79yyEUDVVZA[]a7bb3b>DDQGH')yy':$#%;RXXN^M__`aq`rFsuy=zzz  {; :H4).& :-tDz9 9 +TZZ-A AAcL||kr||zdz}|||kr|dz}n|}||kr|S)Nr*)sortedtargetlowhighmids r# bisect_leftrDgsC *Tza #; 'CD * Jr;c2d}|}||} | ||zz} ||krgt|| ||} t| |dz } ||j||| } || | z }|r| dkDr || z }| ||zz} | |kDr| }n| dz}||krg|dz }||j|||fS)zMerge splits until a chunk size is reached, returning the index of the last split included in the merged chunk along with the merged chunk itself.g?)rArBr*r)rDminjoin)splitscum_lens chunk_sizer" token_counterstartrBaveragerAoffsetr@imidpointtokens local_cumends r# merge_splitsrTts G C e_F zG+ ,F * &c =q$(#x}}VE(-CDEX&/ !&(GzG34F J DQ,C! *$ 'C  fU3/0 00r;c x+|} |} | x} rh|r%tj|t||}|r?|dkrtj||znt ||dz }|r||z } t || } t |\} }}g}t| }|Dcgc] }t|}}tt|d}tdg|Dcgc]}||z c}z}|Dcgc]}||z }}t|dz}g}t+ tt||D]Q\}\}}|+vr||| kDr:t|| |d|dz|\}}|j||j|n_t||| | |||\}}+j!t#|dz||j%||||z }|j%||f|r|t|dz k(rt'+fdt#|dzt|Dr||d| zx} | kr| |d<|d\}}!||!|zf|d<|r|ddn|}|j%| |j%|||zfT| r!t||D"#cgc]\}"}#|"s |"j)r|"|#f}$}"}#|$r"t|$\}}t|t|}}ngg}}|r|r| }%|}&|}'t|&}(tj||%z })tj |%z }*t#t+dtj,|(|)z |*z dzDcgc]'}|'||*zx}d|'t ||)z|(dz df)}}|D!cgc] \}}!|||! }}}!| r||fS|S||fScc}wcc}wcc}wcc}#}"wcc}wcc}!}w) aSplit a text into semantically meaningful chunks of a specified size as determined by the provided token counter. Args: text (str): The text to be chunked. chunk_size (int): The maximum number of tokens a chunk may contain. token_counter (Callable[[str], int]): A callable that takes a string and returns the number of tokens in it. memoize (bool, optional): Whether to memoize the token counter. Defaults to `True`. offsets (bool, optional): Whether to return the start and end offsets of each chunk. Defaults to `False`. overlap (float | int | None, optional): The proportion of the chunk size, or, if >=1, the number of tokens, by which chunks should overlap. Defaults to `None`, in which case no overlapping occurs. cache_maxsize (int | None, optional): The maximum number of text-token count pairs that can be stored in the token counter's cache. Defaults to `None`, which makes the cache unbounded. This argument is only used if `memoize` is `True`. Returns: list[str] | tuple[list[str], list[tuple[int, int]]]: A list of chunks up to `chunk_size`-tokens-long, with any whitespace used to split the text removed, and, if `offsets` is `True`, a list of tuples of the form `(start, end)` where `start` is the index of the first character of the chunk in the original text and `end` is the index of the character after the last character of the chunk such that `chunks[i] == text[offsets[i][0]:offsets[i][1]]`.r*r)initialT)r5rJrKoffsets_recursion_depth_start)rHrIrJr"rKrLrBc3&K|]}|v ywrr>)r!jskipss r#r$zchunk..s'VqU 'Vs)_memoized_token_counters setdefaultrmathfloorrFr:r.r4rset enumeratezipchunkextendrTupdaterangeappendallisspacer,ceil),r5rJrKmemoizerWoverlap cache_maxsizerXrYreturn_offsetslocal_chunk_size is_first_callunoverlapped_chunk_sizer"r6rH splitter_lenr2 split_lensrI split_len split_startsrLnum_splits_plus_onechunksrO split_start new_chunks new_offsetsfinal_split_in_chunk_i new_chunk split_endlast_chunk_with_splitterrSrerNchunks_and_offsets subchunk_size subchunks suboffsets num_subchunkssubchunks_per_chunksubchunk_strider\s, @r#reresc4N!-,,}, 4?? OgyYfOghuOvwM :AA+djjg!563wXbefXfKgG*4w*>'#&w0G#H 0;4/@,H$fGx=L*01#e*1J1Jz156Hqcz$Z)Y%=$ZZ[L0<=uEFN=L=f+/ F EE^$-S-F#G7> E; :   "2 2&+++!1A!5" ' #J  MM* % NN; ' 1=!+!+(1 - "I LLq1u&<= > MM) $%%;<|KI NNK3 4& Vq C'VE!a%VtCFvwCWy-%[`inivivixufoyy !#56OFG"6lDMGF!"GF v,MI J NM#'**]*# #jj'-7Os1diiAT1TXg0g&hkl&lmn  _)< <>qAs5+>#> NQRRSTUVG9@@*%d5o@F@ 7? "  7?c2$Z=Hz2As0N N! ! N& N+N+"N+ ,N1>N6cPeZdZddZ ddZ d ddZy) Chunkerc ||_||_yrrJrK)selfrJrKs r#__init__zChunker.__init__:s$*r;cdfd }|S)zeConstruct a function that chunks a text and returns the chunks along with their offsets if necessary.cNt|jjdS)NF)r5rJrKrmrWrn)rerJrK)r5rWrnrs r#_chunkz,Chunker._make_chunk_function.._chunkEs,??"00  r;)r5strreturn3list[str] | tuple[list[str], list[tuple[int, int]]]r>)rrWrnrs``` r#_make_chunk_functionzChunker._make_chunk_function>s  r;Nc|j||}t|tr||S|r|dk(r t|}|dk(r|Dcgc] }|| }}n4t j |d5} | j |||}ddd|r!t\} } t| t| fSScc}w#1swY3xYw)aSplit text or texts into semantically meaningful chunks of a specified size as determined by the provided tokenizer or token counter. Args: text_or_texts (str | Sequence[str]): The text or texts to be chunked. processes (int, optional): The number of processes to use when chunking multiple texts. Defaults to `1` in which case chunking will occur in the main process. progress (bool, optional): Whether to display a progress bar when chunking multiple texts. Defaults to `False`. offsets (bool, optional): Whether to return the start and end offsets of each chunk. Defaults to `False`. overlap (float | int | None, optional): The proportion of the chunk size, or, if >=1, the number of tokens, by which chunks should overlap. Defaults to `None`, in which case no overlapping occurs. Returns: list[str] | tuple[list[str], list[tuple[int, int]]] | list[list[str]] | tuple[list[list[str]], list[list[tuple[int, int]]]]: If a single text has been provided, a list of chunks up to `chunk_size`-tokens-long, with any whitespace used to split the text removed, and, if `offsets` is `True`, a list of tuples of the form `(start, end)` where `start` is the index of the first character of the chunk in the original text and `end` is the index of the character succeeding the last character of the chunk such that `chunks[i] == text[offsets[i][0]:offsets[i][1]]`. If multiple texts have been provided, a list of lists of chunks, with each inner list corresponding to the chunks of one of the provided input texts, and, if `offsets` is `True`, a list of lists of tuples of the chunks' offsets to the original texts, as described above.)rWrnr*T)use_dill) progress_barN) r isinstancerr mpire WorkerPoolmaprdr4) r text_or_texts processesprogressrWrnchunk_functionr5rpoolryoffsets_s r#__call__zChunker.__call__Qs6227G2T mS )!-0 0  Q /M >CP!Q4."6!Q !Q!!)d; dt%)XXnmZbX%c" d "$67 FH<h/ /!!"R d dsB20B77C)rJintrKCallable[[str], int]rNone)rWboolrnfloat | int | NonerzDCallable[[str], list[str] | tuple[list[str], list[tuple[int, int]]]])r*FFN) rzstr | Sequence[str]rrrrrWrrnzint | float | Nonerz{list[str] | tuple[list[str], list[tuple[int, int]]] | list[list[str]] | tuple[list[list[str]], list[list[tuple[int, int]]]])__name__ __module__ __qualname__rrrr>r;r#rr9sv+$ N ,&* /"*/"/" /"  /" $ /" >/"r;rcZ ttr ddl} |j}|qdD]l}t|stt|s&t|} t| dsD| sGtd| DsZtd| Dntd rhtjt rNjtd rAt#t5t%j'd zdddn td td r9d t)j*j&j,vrdfd } n dfd } n} dz | d fd } | } |r%t.j1| t3|| } t5| S#t$r|j }YwxYw#t$rA ddl}|jj}n#t$rtddwxYwYwxYw#1swYxYw)a/ Construct a chunker that splits one or more texts into semantically meaningful chunks of a specified size as determined by the provided tokenizer or token counter. Args: tokenizer_or_token_counter (str | tiktoken.Encoding | transformers.PreTrainedTokenizer | tokenizers.Tokenizer | Callable[[str], int]): Either: the name of a `tiktoken` or `transformers` tokenizer (with priority given to the former); a tokenizer that possesses an `encode` attribute (e.g., a `tiktoken`, `transformers` or `tokenizers` tokenizer); or a token counter that returns the number of tokens in a input. chunk_size (int, optional): The maximum number of tokens a chunk may contain. Defaults to `None` in which case it will be set to the same value as the tokenizer's `model_max_length` attribute (deducted by the number of tokens returned by attempting to tokenize an empty string) if possible otherwise a `ValueError` will be raised. max_token_chars (int, optional): The maximum number of characters a token may contain. Used to significantly speed up the token counting of long inputs. Defaults to `None` in which case it will either not be used or will, if possible, be set to the number of characters in the longest token in the tokenizer's vocabulary as determined by the `token_byte_values` or `get_vocab` methods. memoize (bool, optional): Whether to memoize the token counter. Defaults to `True`. cache_maxsize (int, optional): The maximum number of text-token count pairs that can be stored in the token counter's cache. Defaults to `None`, which makes the cache unbounded. This argument is only used if `memoize` is `True`. Returns: Callable[[str | Sequence[str], bool, bool, bool, int | float | None], list[str] | tuple[list[str], list[tuple[int, int]]] | list[list[str]] | tuple[list[list[str]], list[list[tuple[int, int]]]]]: A chunker that takes either a single text or a sequence of texts and returns, depending on whether multiple texts have been provided, a list or list of lists of chunks up to `chunk_size`-tokens-long with any whitespace used to split the text removed, and, if the optional `offsets` argument to the chunker is `True`, a list or lists of tuples of the form `(start, end)` where `start` is the index of the first character of a chunk in a text and `end` is the index of the character succeeding the last character of the chunk such that `chunks[i] == text[offsets[i][0]:offsets[i][1]]`. The resulting chunker can be passed a `processes` argument that specifies the number of processes to be used when chunking multiple texts. It is also possible to pass a `progress` argument which, if set to `True` and multiple texts are passed, will display a progress bar. As described above, the `offsets` argument, if set to `True`, will cause the chunker to return the start and end offsets of each chunk. The chunker accepts an `overlap` argument that specifies the proportion of the chunk size, or, if >=1, the number of tokens, by which chunks should overlap. It defaults to `None`, in which case no overlapping occurs.rNrz" was provided to `semchunk.chunkerify` as the name of a tokenizer but neither `tiktoken` nor `transformers` have a tokenizer by that name. Perhaps they are not installed or maybe there is a typo in that name?)token_byte_values get_vocab__iter__c34K|]}t|dyw)__len__N)hasattrr!tokens r#r$zchunkerify..s?m^cy@Y?msc32K|]}t|ywrr.rs r#r$zchunkerify..s)H#e*)Hsmodel_max_lengthencoder+a!Your desired chunk size was not passed to `semchunk.chunkerify` and the provided tokenizer either lacks an attribute named 'model_max_length' or that attribute is not an integer. Either specify a chunk size or provide a tokenizer that has a 'model_max_length' attribute that is an integer.add_special_tokensc<tj|dS)NF)rr.rr5tokenizer_or_token_counters r#rKz!chunkerify..token_counters 5<.token_counters5<.faster_token_countersI"QI4y9$)?EbySbGb@c)dgq)q!A~%)$/ /r;r)r5rrr)rrtiktokenencoding_for_model Exception get_encoding transformers AutoTokenizerfrom_pretrained ValueErrorrcallablegetattrrjr,rrr r.rinspect signature parametersr^r_rr) rrJrrmror tokenizerr$potential_vocabulary_getter_functionvocabrKrrs ``` @r# chunkerifyrsx@,c2   N7H778RS &/"5  013WX]e24XY^b :<`ac5*-%C?mgl?mr;r#rs" 44  9&">S389A]9A4A0(BV 1 1!*18;1GJ1[c1lo1wz11J"& $a aa(a a  a  aaa a9aHG"G"`""& ${G!{G  {G {G{G{G {Gr;