✏️ 正在编辑: model.cpython-312.pyc
路径:
/var/opt/nydus/ops/mysql/ai/ml/__pycache__/model.cpython-312.pyc
提示:
您可以编辑任何文件（包括二进制文件），但请注意不当修改可能导致文件损坏。
� ��'j�l � �� � d Z ddlZddlZddlmZ ddlmZmZmZm Z ddl Z ddl Z ddlmZmZmZmZmZmZmZmZmZmZmZmZmZ ddl m Z G d� de� Z G d � d � Z G d � d e � Z y) z�HeatWave ML model utilities for MySQL Connector/Python. Provides classes to manage training, prediction, scoring, and explanations via MySQL HeatWave stored procedures. � N)�Enum)�Any�Dict�Optional�Union) �VAR_NAME_SPACE�atomic_transaction� convert_to_df� execute_sql�format_value_sql�get_random_name� source_schema�sql_response_to_df�sql_table_from_df�sql_table_to_df� table_exists�temporary_sql_tables� validate_name)�MySQLConnectionAbstractc �P � e Zd ZdZdZdZdZdZdZdZ dZ e d e e d f d e fd �� Zy ) �ML_TASKz/Enumeration of supported ML tasks for HeatWave.�classification� regression� forecasting�anomaly_detection�log_anomaly_detection�recommendation�topic_modeling�task�returnc �> � t | t � r| S | j S )a' Return the string representation of a machine learning task. Args: task (Union[str, ML_TASK]): The task to convert. Accepts either a task enum member (ML_TASK) or a string. Returns: str: The string value of the ML task. )� isinstance�str�value)r s �B/opt/nydus/tmp/pip-target-k0fmikie/lib/python/mysql/ai/ml/model.py�get_task_stringzML_TASK.get_task_stringG s � � �d�C� ��K��z�z�� N)�__name__� __module__� __qualname__�__doc__�CLASSIFICATION� REGRESSION� FORECASTING�ANOMALY_DETECTION�LOG_ANOMALY_DETECTION�RECOMMENDATION�TOPIC_MODELING� staticmethodr r# r& � r' r% r r < sS � �9�%�N� �J� �K�+�� 3��%�N�%�N���e�C��N� 3� �� � �r' r c �H � e Zd ZdZej dfdedeeef de e fd�Z de fd�Z dede e fd �Zde e fd �Zdede fd �Zdd �Zdefd �Zde fd�Zdede e de e ddfd�Zdedede e ddfd�Zdededede e def d�Zdedede e dej2 fd�Zy)�_MyModelCommona8 Common utilities and workflow for MySQL HeatWave ML models. This class handles model lifecycle steps such as loading, fitting, scoring, making predictions, and explaining models or predictions. Not intended for direct instantiation, but as a superclass for heatwave model wrappers. Attributes: db_connection: MySQL connector database connection. task: ML task, e.g., "classification" or "regression". model_name: Identifier of model in MySQL. schema_name: Database schema used for operations and temp tables. N� db_connectionr � model_namec � � || _ t j |� | _ t |� | _ t | j � 5 }t |d� ddd� |�t | j � }t � d|� �| _ | j � d�| _ || _ t |� t | j � 5 }t |d| j � d�|f�� ddd� y# 1 sw Y ��xY w# 1 sw Y yxY w)a� Instantiate _MyMLModelCommon. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-train.html A full list of supported tasks can be found under "Common ML_TRAIN Options" Args: db_connection: MySQL database connection. task: ML task type (default: "classification"). model_name: Name to register the model within MySQL (default: None). Raises: ValueError: If the schema name is not valid DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. Returns: None z(CALL sys.ML_CREATE_OR_UPGRADE_CATALOG();N�.z.scorezSET @z = %s;��params)r7 r r& r r � schema_namer r r �_is_model_name_availabler � model_var�model_var_scorer8 r )�selfr7 r r8 �cursors r% �__init__z_MyModelCommon.__init__i s� � �6 +����+�+�D�1�� �(� �7��� �� 2� 2� 3�v� �� J� K� 4� � �(��)F�)F�G�J� *�+�1�Z�L�9���"&�.�.�!1�� 8�� �$����j�!� �� 2� 2� 3�v� ��%����'7�v� >� �}� U� 4� 3� 4� 3�� 4� 3�s � C%�> C1�%C.�1C:r c �� � | j � }d|� d�}d|� d| j � �}t | j � 5 }t ||� |j dkD cddd� S # 1 sw Y yxY w)a% Deletes the model from the model catalog if present Raises: DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. Returns: Whether the model was deleted � ML_SCHEMA_�.MODEL_CATALOGz DELETE FROM � WHERE model_handle = @r N)� _get_userr? r r7 r �rowcount)rA � current_user�qualified_model_catalog� delete_modelrB s r% � _delete_modelz _MyModelCommon._delete_model� su � � �~�~�'� �$.�|�n�N�"K� ��2�3� 4%�%)�^�^�$4� 6� � �� 2� 2� 3�v� �� � -��?�?�Q�&� 4� 3� 3�s �A#�#A,c � � dt dt fd�}| j � }d|� d�}d|� d�}t | j � 5 }t |||f�� t |� }|j rd }nS|j d � � } t j | � d } | j � D � � ci c] \ } } | || � �� }} } |cd d d � S c c} } w # 1 sw Y y xY w) a� Retrieves the model info from the model_catalog Args: model_var: The model alias to retrieve Returns: The model info from the model_catalog (None if the model is not present in the catalog) Raises: DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. �elemr c � � t | t � r t j | � } | S | S # t j $ r Y | S w xY w�N)r" r# �json�loads�JSONDecodeError)rO s r% � process_colz3_MyModelCommon._get_model_info.<locals>.process_col� sI � ��$��$�� �:�:�d�+�D� �K�4�K�� �+�+� ���K��s �+ �A�ArE rF �SELECT * FROM z WHERE model_handle = %sr; N�records)�orientr ) r rH r r7 r r �empty�to_jsonrR rS �items) rA r8 rU rJ rK � model_existsrB � model_info_df�result�unprocessed_result�unprocessed_result_json�keyrO s r% �_get_model_infoz _MyModelCommon._get_model_info� s� � � �c� �c� � �~�~�'� �$.�|�n�N�"K� � � 4� 5�5M� N� � �� 2� 2� 3�v� �� �j�]� C� .�v� 6�M� �"�"� ��%2�%:�%:�)�%:�%L�"�*.�*�*�5G�*H��*K�'� &=�%B�%B�%D��%D� ��T� ��T�*�*�%D� � � � 4� 3��� 4� 3�s �A'C�)C �<C� C�Cc �8 � | j | j � S )a� Checks if the model name is available. Model info is present in the catalog only if the model was previously fitted. Returns: True if the model name is not part of the model catalog Raises: DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. )rb r8 )rA s r% �get_model_infoz _MyModelCommon.get_model_info� s � � �#�#�D�O�O�4�4r' c �( � | j |� du S )a1 Checks if the model name is available Returns: True if the model name is not part of the model catalog Raises: DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. N)rb )rA r8 s r% r> z'_MyModelCommon._is_model_name_available� s � � �#�#�J�/�4�7�7r' c � � t | j � 5 }d| j � d�}t ||� ddd� y# 1 sw Y yxY w)a Loads the model specified by `self.model_name` into MySQL. After loading, the model is ready to handle ML operations. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-model-load.html Raises: DatabaseError: If the model is not initialized, i.e., fit or import has not been called If a database connection issue occurs. If an operational error occurs during execution. Returns: None zCALL sys.ML_MODEL_LOAD(@z, NULL);N)r r7 r? r )rA rB �load_model_querys r% � _load_modelz_MyModelCommon._load_model� s? � �" �� 2� 2� 3�v�!9�$�.�.�9I�� R� � �� 0� 1� 4� 3� 3�s � <�Ac �� � t | j � 5 }|j d� |j � d j d� d }t |� cddd� S # 1 sw Y yxY w)a� Fetch the current database user (without host). Returns: The username string associated with the connection. Raises: DatabaseError: If a database connection issue occurs. If an operational error occurs during execution. ValueError: If the user name includes unsupported characters zSELECT CURRENT_USER()r �@N)r r7 �execute�fetchone�splitr )rA rB rJ s r% rH z_MyModelCommon._get_user sV � � �� 2� 2� 3�v� �N�N�2� 3�!�?�?�,�Q�/�5�5�c�:�1�=�L� � �.� 4� 3� 3�s �AA!�!A*c � � | j � t | j � 5 }| j � }d|� d�}d|� d| j � �}t ||� t |� }|j d cddd� S # 1 sw Y yxY w)a� Get model explanations, such as detailed feature importances. Returns: dict: Feature importances and model explainability data. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-model-explanations.html Raises: DatabaseError: If the model is not initialized, i.e., fit or import has not been called If a database connection issue occurs. If an operational error occurs during execution. ValueError: If the model does not exist in the model catalog. Should only occur if model was not fitted or was deleted. rE rF z SELECT model_explanation FROM rG �r r N)rh r r7 rH r? r r �iloc)rA rB rJ rK � explain_query�dfs r% � explain_modelz _MyModelCommon.explain_model# s� � �& ���� �� 2� 2� 3�v� �>�>�+�L�(2�<�.��&O� #�0�1H�0I� J)�)-� � �(8�:� � �� � .�#�F�+�B��7�7�4�=� 4� 3� 3�s �AA>�>B� table_name�target_column_name�optionsc � � t |� |�t |� d|� d�}nd}|�i }t j |� }| j |d< | j � t | j � 5 }t |� \ }}t |d| j � d|� d|� d|� d | j � d � |� � ddd� y# 1 sw Y yxY w) a2 Fit an ML model using a referenced SQL table and target column. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-train.html A full list of supported options can be found under "Common ML_TRAIN Options" Args: table_name: Name of the training data table. target_column_name: Name of the target/label column. options: Additional fit/config options (may override defaults). Raises: DatabaseError: If provided options are invalid or unsupported. If a database connection issue occurs. If an operational error occurs during execution. ValueError: If the table or target_column name is not valid Returns: None N�'�NULLr zCALL sys.ML_TRAIN('r: �', �, z, @�)r; ) r �copy�deepcopyr rM r r7 r r r= r? )rA rt ru rv �target_col_stringrB � placeholders� parameterss r% �_fitz_MyModelCommon._fitE s� � �8 �j�!� � )� �,� -�"#�$6�#7�q� 9� � &� � �?��G��-�-��(�� �)�)���� ��� � �� 2� 2� 3�v�'7��'@� $�L�*� ��� �(�(�)��:�,�c�(�)��#�n� %� ���'�� � "� � 4� 3� 3�s �/AB9�9C�output_table_namec �> � t |� t |� | j � t | j � 5 }t |� \ }}t |d| j � d|� d| j � d| j � d|� d|� d� |�� ddd� y# 1 sw Y yxY w) a� Predict on a given data table and write results to an output table. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-predict-table.html A full list of supported options can be found under "ML_PREDICT_TABLE Options" Args: table_name: Name of the SQL table with input data. output_table_name: Name for the SQL output table to contain predictions. options: Optional prediction options. Returns: None Raises: DatabaseError: If provided options are invalid or unsupported, or if the model is not initialized, i.e., fit or import has not been called If a database connection issue occurs. If an operational error occurs during execution. ValueError: If the table or output_table name is not valid zCALL sys.ML_PREDICT_TABLE('r: �', @�, 'rz r| r; N)r rh r r7 r r r= r? )rA rt r� rv rB r� r� s r% �_predictz_MyModelCommon._predict~ s� � �6 �j�!��'�(� ���� �� 2� 2� 3�v�'7��'@� $�L�*� ��� �(�(�)��:�,� 7� ���'� (� �(�(�)��+<�*=�S�#�n�� � "� � 4� 3� 3�s �AB�B �metricc � � t |� t |� t |� | j � t | j � 5 }t |� \ }}t |d| j � d|� d|� d| j � d| j � d|� d� |g|��� t |d | j � �� t |� }|j d cd d d � S # 1 sw Y y xY w) aI Evaluate model performance with a scoring metric. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-score.html A full list of supported options can be found under "Options for Recommendation Models" and "Options for Anomaly Detection Models" Args: table_name: Table with features and ground truth. target_column_name: Column of true target labels. metric: String name of the metric to compute. options: Optional dictionary of further scoring options. Returns: float: Computed score from the ML system. Raises: DatabaseError: If provided options are invalid or unsupported, or if the model is not initialized, i.e., fit or import has not been called If a database connection issue occurs. If an operational error occurs during execution. ValueError: If the table or target_column name or metric is not valid zCALL sys.ML_SCORE('r: z', 'r� z, %s, @r{ r| r; zSELECT @ro N) r rh r r7 r r r= r? r@ r rp ) rA rt ru r� rv rB r� r� rr s r% �_scorez_MyModelCommon._score� s� � �D �j�!��(�)��f� � ���� �� 2� 2� 3�v�'7��'@� $�L�*� ��� �(�(�)��:�,� 7�*�+� ,� ���'� (� �,�,�-�R�#�n��� �,��,� � ��(�4�+?�+?�*@� A� B�#�F�+�B��7�7�4�=�' 4� 3� 3�s �BC�C c � � t |� t |� |�ddi}| j � t | j � 5 }t |� \ }}t |d| j � d|� d| j � d| j � d|� d|� d � |� � t |d | j � d|� �� t |� }|cddd� S # 1 sw Y yxY w) a Produce explanations for model predictions on provided data. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-explain-table.html A full list of supported options can be found under "ML_EXPLAIN_TABLE Options" Args: table_name: Name of the SQL table with input data. output_table_name: Name for the SQL table to store explanations. options: Optional dictionary (default: {"prediction_explainer": "permutation_importance"}). Returns: DataFrame: Prediction explanations from the output SQL table. Raises: DatabaseError: If provided options are invalid or unsupported, or if the model is not initialized, i.e., fit or import has not been called If a database connection issue occurs. If an operational error occurs during execution. ValueError: If the table or output_table name is not valid N�prediction_explainer�permutation_importancezCALL sys.ML_EXPLAIN_TABLE('r: r� r� rz r| r; rV ) r rh r r7 r r r= r? r )rA rt r� rv rB r� r� rr s r% �_explain_predictionsz#_MyModelCommon._explain_predictions� s� � �8 �j�!��'�(� �?�-�/G�H�G� ���� �� 2� 2� 3�v�'7��'@� $�L�*� ��� �(�(�)��:�,� 7� ���'� (� �(�(�)��+<�*=�S�#�n�� � "� � ��.��1A�1A�0B�!�DU�CV� W� X�#�F�+�B��# 4� 3� 3�s �A6C�C )r N)r( r) r* r+ r r, r r r# r rC �boolrM �dictrb rd r> rh rH rs r� r� �floatr� �pd� DataFramer� r4 r' r% r6 r6 Z sr � � �" %,�$:�$:�$(� ,V�.�,V� �C��L�!�,V� �S�M� ,V�\'�t� '�0-�#� -�(�4�.� -�^ 5���� 5� 8�3� 8�4� 8� 2�*/�3� /�& !�t� !�D7��7� %�S�M�7� �$� � 7� � 7�r,� �,�25�,�@H���,� �,�\:!��:!� �:!� � :!� �$� � :!� � :!�x5� �5�25�5�@H���5� � � �5r' r6 c �� � e Zd ZdZ d deej ej f de eej ej f de e ddfd�Z d deej ej f de e dej fd�Z d deej ej f deej ej f d e de e def d �Z d deej ej f dedej fd �Zy) �MyModelz� Convenience class for managing the ML workflow using pandas DataFrames. Methods convert in-memory DataFrames into temp SQL tables before delegating to the _MyMLModelCommon routines, and automatically clean up temp resources. N�X�yrv r c �V �� t �� t |� c�}t | j � 5 }t | j � 5 }|�mt |t j � r|j d }nt �fd�� }|�j v rt d|� d�� ��j � }|||< |}nd}�}t || j |� \ } } |j | j | f� | j | ||� ddd� ddd� y# 1 sw Y �xY w# 1 sw Y yxY w)a� Fit a model using DataFrame inputs. If an 'id' column is defined in either dataframe, it will be used as the primary key. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-train.html A full list of supported options can be found under "Common ML_TRAIN Options" Args: X: Features DataFrame. y: (Optional) Target labels DataFrame or Series. If None, only X is used. options: Additional options to pass to training. Returns: None Raises: DatabaseError: If provided options are invalid or unsupported. If a database connection issue occurs. If an operational error occurs during execution. Notes: Combines X and y as necessary. Creates a temporary table in the schema for training, and deletes it afterward. Nr c � �� | �j vS rQ ��columns��namer� s �r% �<lambda>z MyModel.fit.<locals>.<lambda>U s �� �T����%:r' zTarget column y with name z' already present in feature dataframe X)r r r7 r r" r� r� r� r � ValueErrorr} r r= �appendr� ) rA r� r� rv rB �temporary_tablesru � df_combined�final_df�_rt s ` r% �fitz MyModel.fit( s# �� �B �Q� � �q�!1���1� �t� 1� 1� 2�f� ��!3�!3� 4�8H��}� �a�� � �.�)*���1� �&�)8�:�*�&� &����2�$�4�5G�4H� I1� 1�� � �f�f�h� �23� � .�/�&��%)�"� �� -�f�d�6F�6F�� Q�M�A�z� � #� #�T�%5�%5�z�$B� C� �I�I�j�"4�g� >�7 5� 3� 2� 4� 4�� 3� 2�s$ �D �B>D�D �D �D � D(c �D � �� t |� }t � j � 5 �t � j � 5 }t �� j |� \ }}|j � j |f� t �� fd�� }|j � j |f� � j |||� t �� j |� }|d j t j � |d< |cddd� cddd� S # 1 sw Y nxY w ddd� y# 1 sw Y yxY w)a� Generate model predictions using DataFrame input. If an 'id' column is defined in either dataframe, it will be used as the primary key. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-predict-table.html A full list of supported options can be found under "ML_PREDICT_TABLE Options" Args: X: DataFrame containing prediction features (no labels). options: Additional prediction settings. Returns: DataFrame with prediction results as returned by HeatWave. Raises: DatabaseError: If provided options are invalid or unsupported, or if the model is not initialized, i.e., fit or import has not been called If a database connection issue occurs. If an operational error occurs during execution. Notes: Temporary SQL tables are created and deleted for input/output. c �4 �� t ��j | � S rQ �r r= �rt rB rA s ��r% r� z!MyModel.predict.<locals>.<lambda>� � �� �|��D� ,� ,�j�(� $r' � ml_resultsN) r r r7 r r r= r� r r� r �maprR rS ) rA r� rv r� r� rt r� � predictionsrB s ` @r% �predictzMyModel.predictj s� �� �@ �!� �� �t� 1� 1� 2�f� ��!3�!3� 4�8H� -�f�d�6F�6F�� J�M�A�z� � #� #�T�%5�%5�z�$B� C� /��!� � � #� #�T�%5�%5�7H�$I� J� �M�M�*�&7�� A�)�&�$�2B�2B�DU�V�K� )4�L�(A�(E�(E�d�j�j�(Q�K� � %� �% 5� 4� 3� 2� 4� 4�� 4� 3� 2� 2�s# �D�B4D �- D� D �D�D r� c �� �� t �� t |� c�}t | j � 5 }t | j � 5 }t �fd�� }�j � }|||< |} t || j | � \ } } |j | j | f� | j | |||� } | cddd� cddd� S # 1 sw Y nxY w ddd� y# 1 sw Y yxY w)a Score the model using X/y data and a selected metric. If an 'id' column is defined in either dataframe, it will be used as the primary key. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-score.html A full list of supported options can be found under "Options for Recommendation Models" and "Options for Anomaly Detection Models" Args: X: DataFrame of features. y: DataFrame or Series of labels. metric: Metric name (e.g., "balanced_accuracy"). options: Optional ml scoring options. Raises: DatabaseError: If provided options are invalid or unsupported, or if the model is not initialized, i.e., fit or import has not been called If a database connection issue occurs. If an operational error occurs during execution. Returns: float: Computed score. c � �� | �j vS rQ r� r� s �r% r� z MyModel.score.<locals>.<lambda>� s �� �d�!�)�)�>Sr' N) r r r7 r r r} r r= r� r� ) rA r� r� r� rv rB r� ru r� r� r� rt �scores ` r% r� z MyModel.score� s� �� �F �Q� � �q�!1���1� �t� 1� 1� 2�f� ��!3�!3� 4�8H�!0�1S�!T� ��&�&�(�K�./�K�*� +�"�H� -�f�d�6F�6F�� Q�M�A�z� � #� #�T�%5�%5�z�$B� C��K�K� �,>���P�E�� 5� 4� 3� 2� 4� 4�� 4� 3� 2� 2�s$ �C �A2C �6 C � C �C � C(c �� � �� t |� }t � j � 5 �t � j � 5 }t �� j |� \ }}|j � j |f� t �� fd�� }|j � j |f� � j |||� }|cddd� cddd� S # 1 sw Y nxY w ddd� y# 1 sw Y yxY w)a Explain model predictions using provided data. If an 'id' column is defined in either dataframe, it will be used as the primary key. References: https://dev.mysql.com/doc/heatwave/en/mys-hwaml-ml-explain-table.html A full list of supported options can be found under "ML_EXPLAIN_TABLE Options" Args: X: DataFrame for which predictions should be explained. options: Optional dictionary of explainability options. Returns: DataFrame containing explanation details (feature attributions, etc.) Raises: DatabaseError: If provided options are invalid or unsupported, or if the model is not initialized, i.e., fit or import has not been called If a database connection issue occurs. If an operational error occurs during execution. Notes: Temporary input/output tables are cleaned up after explanation. c �4 �� t ��j | � S rQ r� r� s ��r% r� z-MyModel.explain_predictions.<locals>.<lambda> r� r' N) r r r7 r r r= r� r r� ) rA r� rv r� r� rt r� � explanationsrB s ` @r% �explain_predictionszMyModel.explain_predictions� s� �� �@ �!� �� �t� 1� 1� 2�f� ��!3�!3� 4�8H� .�f�d�6F�6F�� J�M�A�z� � #� #�T�%5�%5�z�$B� C� /��!� � � #� #�T�%5�%5�7H�$I� J� �4�4�� -�w� �L� �# 5� 4� 3� 2� 4� 4�� 4� 3� 2� 2�s# �C�A8C�1 C�C � C�C#rQ )r( r) r* r+ r r� r� �np�ndarrayr r� r� r� r# r� r� r r� r4 r' r% r� r� sJ � �� #'� @?� �� � �r�z�z�)� *�@?� �E�"�,�,�� � �2�3� 4�@?� �$� � @?� � @?�J #'�6 � �� � �r�z�z�)� *�6 � �$� �6 � � � � 6 �z #'� 3� �� � �r�z�z�)� *�3� �� � �r�z�z�)� *�3� � 3� �$� � 3� � 3�p �5 � �� � �r�z�z�)� *�5 � �5 � � � � 5 r' r� )!r+ r} rR �enumr �typingr r r r �numpyr� �pandasr� �mysql.ai.utilsr r r r r r r r r r r r r �mysql.connector.abstractsr r r6 r� r4 r' r% �<module>r� sa ��8� � � � -� -� � �� � � � >��d� �<C� C�Ll �n� l r'
💾 保存文件
← 返回文件管理器