Ripping off RouteLLM

The context

RouteLLM, a project by LMSYS and Anyscale, is designed to route queries between LLMs. They trained models to convert query embeddings into performance scores, using various data sources including human preferences and GPT-based scoring. This lets you route between LLMs based on the query's difficulting, improving the quality of responses while saving costs.

I have a project where children can chat with AI (thanks to Groq's free services). The main LLM is Gemma 2 9b, but it lets users choose smarter ones if they want. The conundrum I faced was that Llama 3 70b was definitely better in some cases, but not all, and I needed a router to decide when to use it. RouteLLM let me do that.

The goal

Inspired by RouteLLM's models, I figured I could create vectors that could act as routers. These vectors, when dotted with a prompt's embedding, output a LLM's expected performance.

How

Step 1: training data

Take RouteLLM's GPT-augmented dataset. Make a new one that includes the prompt, the LLM, and the LLM's score. (This was done by sampling challenging prompts from the dataset and comparing the reference answer to each model's response.)

Step 2: train in data

Turn the dataset into one dataset per LLM, where the input is the prompt's embedding and the output is the LLM's score. Get gradient descent code from GPT so I can train a "model" - a vector that can be dotted with the embedding - on this. Success.

Step 3: post the results online

Actual vectors
Gemma:
[0.077, -0.11, 0.526, 0.581, -0.377, -0.493, -0.138, 0.695, -0.133, 0.025, 0.385, -0.321, -0.49, -0.257, 0.396, 0.039, -0.116, 0.261, -0.407, 0.766, 0.32, 0.645, -0.006, 0.11, 0.27, -0.255, -0.202, 0.067, 0.384, 0.221, 0.542, -1.6, -0.5, -0.526, -0.066, 0.022, 0.513, 0.062, 0.289, 0.085, 0.007, -0.273, 0.653, 0.33, -0.208, 0.155, -0.154, -0.295, 0.173, 0.327, -0.389, 0.028, -0.045, 0.305, -0.111, -0.211, -0.188, 0.266, 0.089, 0.154, 0.538, -0.387, 0.003, 0.254, -0.56, 0.214, -0.189, 0.531, 0.058, 0.339, -0.146, -0.015, 0.246, -0.107, 0.032, 0.149, -0.177, 0.133, 0.341, 0.543, -0.169, 0.116, -0.562, 0.088, -0.791, -0.383, -0.628, 0.162, -0.233, -0.518, -0.053, 0.647, -0.424, 0.425, 0.508, 0.063, 0.208, 0.343, -0.074, 0.157, 0.152, -0.575, -0.095, 0.164, 0.675, -0.016, 0.177, -0.055, -0.278, 0.071, -0.59, -0.25, -0.156, 0.05, -0.194, -0.094, -0.23, -0.466, 0.004, -0.395, -0.376, -0.006, 0.351, -0.214, -0.182, -0.125, 0.035, -0.232, -0.123, -0.569, 0.455, -0.679, 0.779, -0.717, -0.587, 0.157, -0.142, 0.121, -0.709, 0.529, 0.294, -0.09, 0.121, 0.201, 0.109, -0.392, 0.029, 0.162, -0.378, -0.12, -0.369, 0.07, -0.49, 0.239, -0.192, -0.549, 0.483, 0.137, -0.373, 0.487, -0.665, 0.522, 0.043, -0.223, -0.495, 0.568, -0.265, -0.442, -0.645, 0.02, -0.356, 0.077, 0.443, -0.088, 0.316, -0.185, 0.057, -0.4, -0.283, -0.309, -0.235, 0.672, 0.007, -0.335, -0.188, 0.243, 0.004, 0.185, -0.523, 0.253, -0.428, 0.495, 0.31, 0.167, 0.08, -0.902, 0.592, 0.244, 0.377, 0.017, -0.261, 0.086, -0.04, -0.402, -0.23, -0.417, -0.158, 0.194, 0.038, -0.141, 0.407, -0.482, -0.053, 0.585, -0.356, -0.103, -0.32, 0.794, 0.415, 0.436, 0.068, 0.201, 0.755, 0.753, -0.173, 0.566, 0.873, -0.064, 0.192, -0.169, -0.082, 0.157, -0.536, -0.378, -0.174, 0.107, -0.449, -0.17, 0.328, -0.145, -0.381, -0.355, -0.193, 0.401, -0.574, 0.176, -0.529, -0.902, -0.238, 0.504, 0.133, -0.096, 0.766, -0.057, -0.658, 0.156, 0.199, -0.035, 0.067, -0.455, -0.204, -0.128, -0.639, -0.537, -0.601, -0.281, -0.28, -0.217, -0.258, 0.149, 0.286, -0.364, 0.367, -0.28, -0.416, 0.151, 0.287, 0.039, 0.195, 0.287, -0.602, -0.339, 0.067, 0.03, 0.134, 0.383, 0.394, 0.057, 0.206, -0.416, 0.265, -0.488, 0.043, -0.055, 0.234, -0.421, -0.269, 0.098, -0.016, 0.273, 0.672, 0.212, -0.161, 0.227, -0.471, 0.27, 0.143, -0.121, 0.286, 0.166, -0.131, 0.134, 0.217, -0.17, 0.127, -0.755, 0.243, 0.143, 0.183, -0.719, 0.203, -0.925, 0.194, 0.297, -0.323, -0.263, 0.201, -0.219, 0.126, -0.505, -0.596, -0.267, -0.17, 0.078, -0.027, -0.266, -0.22, 0.113, 0.018, -0.386, -0.74, -0.608, -0.08, 0.388, -0.105, 0.014, 0.143, 0.4, 0.067, 0.289, -0.155, 0.153, 0.169, 0.225, 0.267, -0.013, 0.592, -0.447, -0.219, -0.164, -0.246, 0.019, 0.138, 0.581, -0.171, 0.474, -0.224, -0.179, 0.272, 0.143, -0.372, 0.274, -0.364, 0.399, -0.623, 0.568, 0.407, 0.521, 0.598, 0.196, 0.221, 0.555, -0.055, -0.688, 0.431, 0.311, 0.392, -0.479, -0.688, -0.537, 0.501, -0.513, 0.531, 0.224, -0.269, -0.076, 0.59, 0.913, -0.474, -0.253, 0.217, -0.359, -0.37, -0.376, 0.22, 0.403, -0.602, -0.306, 0.148, -0.269, 0.249, -0.534, 1.051, 0.033, 0.399, -0.253, 0.112, -0.134, 0.054, 0.319, 0.094, -0.164, -0.081, -0.133, 0.08, 0.485, 0.362, -0.624, -0.062, -1.081, 0.208, 0.264, 0.209, 0.237, 0.033, -0.205, -0.198, 0.477, 0.235, 0.146, 0.16, 0.445, -0.207, -0.108, -0.712, 0.168, 0.257, 0.318, 0.022, -0.384, -0.045, -0.459, -0.332, -0.137, 0.085, 0.307, 0.443, 0.139, 0.352, 0.197, 0.285, -0.17, -0.122, 0.197, 0.173, 0.091, 0.321, 0.598, 0.035, -0.559, 0.056, -0.092, 0.205, 0.223, -0.16, 0.287, 0.816, 0.104, 0.227, -0.615, -0.179, 0.052, -0.091, -0.181, -0.049, 0.534, 0.1, -0.015, -0.937, -0.306, -0.332, 0.508, 0.751, -0.592, 0.068, -0.305, -0.564, -0.584, -0.031, -0.37, 0.513, 0.219, -0.298, 0.755, 0.142, 0.027, -0.204, -0.049, 0.163, 0.445, -0.444, 0.085, 0.135, -0.281, -0.423, 0.429, 0.084, 0.509, -0.271, -0.369, -0.263, 0.174, -0.057, -0.041, 0.226, -0.437, -0.579, -0.199, -0.516, -0.386, 0.541, -0.683, -0.25, 0.148, -0.268, -0.175, -0.115, 0.444, -0.187, -0.204, 0.085, 0.253, 0.298, -0.37, -0.432, -0.016, -0.315, -0.023, 0.174, 0.4, 0.223, 0.094, -0.179, -0.388, 0.366, -0.152, -0.16, -0.341, 0.183, -0.136, -0.044, -0.018, 0.479, -0.111, -0.176, 0.487, -0.176, -0.182, 0.096, -0.023, -0.373, -0.074, -0.342, -1.01, 0.438, 0.384, 0.522, -0.318, 0.238, 0.114, -0.004, -0.295, -0.279, -0.345, -0.481, 0.334, 0.366, -0.173, 0.27, -0.029, 0.335, 0.247, 0.258, 0.146, -0.142, 0.029, -0.23, -0.185, -0.445, 0.163, -0.07, 0.04, 0.217, -0.304, 0.206, 0.523, -0.16, -0.197, 0.07, -0.062, 0.006, 0.384, -0.296, -0.829, 0.166, 0.05, -0.381, 0.155, -0.014, -0.11, 0.152, -0.224, -0.208, 0.311, 0.134, 0.299, -0.097, -0.527, -0.18, 0.323, 0.375, -0.149, 0.581, 0.103, -0.144, -0.048, 0.154, 0.035, -0.08, -0.386, 0.191, -0.041, 0.184, -0.269, 0.526, -0.047, -0.385, 0.141, 0.289, -0.273, -0.241, -0.024, -0.04, -0.046, 0.272, 0.305, -0.335, -0.196, -0.402, 0.069, -0.129, -0.037, -0.064, -0.05, 0.225, 0.159, -0.018, 0.195, 0.134, 0.118, -0.204, 0.725, 0.081, 0.051, 0.168, 0.375, 0.079, -0.464, -0.278, 0.27, 0.027, 0.605, -0.321, -0.152, -0.338, 0.097, 0.424, 0.079, -0.106, -0.121, -0.555, 0.137, 0.525, 0.237, 0.493, -0.157, 0.261, 0.118, 0.088, -0.66, 0.161, 0.364, -0.07, -0.443, -0.257, -0.05, -0.078, 0.31, 0.449, 0.069, 0.231, -0.029, -0.171, -0.02, -0.068, 0.147, -0.011, 0.125, 0.15, -0.25, -0.129, 0.379, -0.335, -0.556, 0.468, 0.042, -0.33, -0.128, -0.294, 0.183, -0.219, -0.126, 0.33, 0.546, 0.214, -0.175, -0.52, -0.247, -0.582, -0.646, -0.264, -0.075, -0.052, 0.227, 0.43, 0.203, 0.385, 0.414, -0.162, -0.38, -0.362, 0.35, 0.09, 0.22, -0.142, -0.954, -0.01, -0.296, 0.358, 0.515, 0.15, 0.54, -0.361, -0.37, -0.099, 0.174, 0.001, -0.211, 0.044, -0.209, -0.03, -0.683, 0.004, -0.153, 0.267, -0.055, -0.358, -0.056, 0.09, 0.202, 0.165, -0.684, 0.116, 0.293, -0.425, -0.183, 0.205, -0.009, -0.171, -0.692, -0.523, 0.216, -0.251, 0.171, 0.38, -0.052, -0.104, -1.852, -0.357, 0.197, 0.054, -0.243, 0.07, 0.149, 0.465, 0.288, -0.248, 0.115, -0.3, -0.27, 0.252, 0.488, -0.47, -0.015, 0.241, -0.35, 0.31, -0.359, 0.176, -0.046, 0.165, 0.08, 0.167, -0.292, 0.265, -0.27, 0.072, 0.275, 0.229, -0.364, 0.129, 0.086, -0.255, 0.197, 0.023, -0.105, 0.056, -0.22, -0.363, -0.094, 0.043, 0.084, -0.054, 0.07, 0.046, 0.002, 0.168, 0.362, 0.027, 0.104, -0.355, 0.166, 0.353, -0.108, -0.135, 0.197, 0.119, 0.137, -0.326, -0.505, -0.368, -0.284, -0.294, -0.256, 0.234, -0.516, -0.136, -0.098, 0.064, -0.09, 0.069, 0.039, -0.552, -0.223, 0.358, -0.04, -0.253, 0.301, -0.022, -0.22, -0.002, 0.186, -0.062, 0.001, -0.106, -0.169, -0.087, 0.827, 0.187, 0.398, 0.502, 0.023, 0.235, -0.43, -0.017, 0.239, -0.009, 0.474, 0.028, -0.064, -0.203, -0.178, -0.022, 0.477, 0.086, 0.213, 0.003, 0.484, -0.124, -0.046, -0.031, -0.522, 0.293, 0.332, 0.538, 0.387, 0.261, 0.656, -0.149, -0.379, -0.158, 0.224, -0.068, -0.016, 0.24, 0.212, -0.294, 0.291, 0.63, -0.118, 0.185, 0.318, -0.174, 0.274, 0.573, -0.094, 0.044, -0.052, -0.238, 0.054, 0.376, -0.313, 0.102, -0.11, 0.104, 0.078, -0.276, 0.139, -0.035, -0.168, 0.561, -0.359, 0.534, 0.002, 0.41, -0.16, 0.436, 0.008, 0.47, -0.034, -0.173, 0.309, 0.292, 0.14, -0.079, 0.462, -0.007, 0.065, 0.128, 0.027, -0.234, 0.175, 0.152, 0.35, -0.44, 0.02, 0.351, 0.309, 0.197, 0.045, 0.357, 0.647, -0.004, 0.234, 0.268, 0.727, 0.419, 0.228, 0.086, 0.115, 0.164, 0.135, -0.08, 0.258, 0.176, 0.076, -0.233, -0.809, 0.008, 0.427, -0.108, -0.147, 0.183, 0.101, -0.409, -0.181, 0.166, -0.252, -0.35, -0.354, 0.191, 0.291, 0.045, 0.025, -0.199, -0.584, 0.358, -0.299, -0.617, 0.018, 0.342, -0.256, 0.236, 0.033, -0.386, 0.326, 0.398, -0.008, 1.019, 0.516, 0.335, 0.093, -0.358, 0.115, 0.073, 0.733, -0.052, -0.397, -0.583, 0.332, -0.492, -0.177, -0.1, 0.24, -0.511, 0.463, -0.348, -0.07, 0.25, 0.243, -0.17, -0.001, 0.096, -0.151, 0.277, 0.186, 0.104, -0.234, -0.108, -0.016, -0.024, -0.402, -0.089, -0.423, 0.231, 0.048, -0.313, -0.278, -0.127, 0.327, 0.367, 0.12, 0.276, 0.447, -0.413, -0.026, -0.41, -0.335, 0.458, 0.423, 0.363, 0.214, 0.161, -0.146, -0.23, 0.027, 0.122, -0.214, -0.071, -0.363, -0.33, -0.121, -0.319, -0.205, 0.046, 0.358, 0.296, 0.559, -0.056, 0.01, 0.031, 0.231, -0.273, -0.262, 0.036, -0.457, -0.431, 0.077, 0.16, -0.067, 0.115, 0.154, -0.066, 0.188, -0.039, 0.321, -0.091, -0.283, 0.146, -0.217, -0.639, -0.117, -0.22, 0.212, 0.056, -0.073, 0.209, -0.177, 0.064, -0.032, -0.27, -0.268, 0.168, -0.076, 0.138, -0.102, 0.447, -0.394, -0.364, 0.297, 0.578, -0.055, -0.308, 0.459, 0.328, 0.044, 0.085, 2.947, -0.205, -0.14, -0.012, 0.153, 0.311, -0.36, -0.016, -0.216, 0.256, -0.291, -0.179, 0.287, -0.047, -0.258, -0.139, -0.185, 0.034, -0.099, -0.201, -0.391, -0.381, -0.457, 0.313, -0.146, -0.207, 0.189, 0.056, 0.26, -0.017, 0.148, 0.24, 0.109, 0.221, -0.211, 0.367, -0.446, 0.01, 0.274, 0.356, -0.216, 0.307, 0.362, -0.279, 0.159, 0.004, -0.243, -0.195, -0.115, -0.036, -0.328, -0.137, 0.005, 0.265, 0.004, 0.119, -0.148, -0.073, 0.894, 0.31, 0.214, -0.315, -0.322, -0.027, -0.037, 0.382, -0.454, -0.254, 0.035, -0.063, 0.064, 0.065, -0.641, -0.104, 0.332, 0.377, 0.191, -0.529, 0.137, -0.732, -0.484, -0.229, 0.412, 0.16, 0.214, -0.499, -0.328, 0.334, 0.238, 0.336, -0.051, 0.29, 0.235, -0.364, 0.036, 0.216, 0.001, 0.157, -0.11, 0.191, 0.156, 0.022, -0.515, 0.38, -0.109, -0.412, 0.172, -0.361, -0.226, -0.047, 0.455, -0.122, -0.094, -0.364, 0.217, -0.147, -0.116, 0.047, 0.067, -0.334, -0.084, 0.265, 0.363, -0.36, 0.22, 0.235, 0.36, 0.353, 0.06, -0.203, 0.339, 0.418, 0.498, 0.264, 0.598, 0.215, 0.042, -0.076, 0.324, 0.019, -0.358, 0.152, -0.014, -0.614, 0.162, 0.284, -0.176, 0.025, 0.012, 0.053, -0.01, 0.336, -0.399, 0.177, 0.388, 0.06, -0.218, -0.155, -0.504, -0.043, -0.249, 0.175, 0.12, -0.122, 0.429, 0.136, 0.007, -0.043, -0.607, -0.204, 0.344, 0.145, -0.092, 0.12, 0.279, -0.272, -0.202, 0.384, -0.246, 0.048, -0.237, -0.346, 0.054, 0.13, -0.146, 0.094, -0.281, -0.53, -0.004, 0.037, 0.012, -0.183, 0.287, -0.05, -0.628, 0.414, 0.143, -0.013, -0.023, -0.04, -0.07, -0.121, -0.019, 0.112, -0.177, -0.167, 0.612, 0.224, 0.097, 0.166, 0.308, -0.089, 0.166, 0.626, 0.203, 0.501, 0.055, 0.283, 0.158, -0.185, -0.125, -0.183, -0.025, -0.019, -0.442, -0.136, 0.107, -0.382, -0.014, 0.184, 0.177, -0.073, 0.321, -0.265, 0.087, 0.25, -0.671, -0.234, -0.022, 0.315, -0.142, -0.07, -0.339, 0.03, -0.117, 0.37, 0.665, -0.71, 0.16, 0.277, 0.338, 0.128, -0.018, -0.078, -0.398, 0.42, 0.166, 0.066, 0.123, -0.212, 0.153, 0.371, 0.282, 0.118, -0.358, 0.081, -0.589, -0.329, -0.353, 0.183, -0.409, 0.16, -0.218, -0.138, -0.594, 0.092, 0.033, 0.422, 0.598, -0.072, -0.164, -0.3, -0.302, -0.308, 0.091, -0.261, 0.017, 0.49, -0.35, -0.338, 0.37, -0.058, -0.198, 0.202, 0.314, 0.215, 0.036, -0.175, -0.193, 0.068, 0.0, 0.105, -0.344, -0.088, 0.192, 0.551, -0.115, 0.468, -0.1, 0.03, 0.32, -0.388, -0.189, 0.045, -0.204, 0.34, 0.005, 0.21, 0.023, 0.477, 0.447, 0.193, 0.417, 0.227, 0.441, -0.141, -0.287, 0.025, 0.243, -0.273, 0.357, 0.002, 0.462, -0.038, 0.421, -0.434, -0.25, 0.232, 0.065, -0.691, -0.214, 0.163, -0.002, 0.533, -0.233, -0.042, 0.302, 0.301, 0.034, 0.011, 0.33, 0.016, -0.307, -0.112, -0.276, 0.393, 0.597, -0.338, -0.12, 0.119, -0.146, -0.146, -0.272, 0.461, 0.081, 0.102, -0.451, -0.062, -0.202, 0.45, -0.357, 0.316, 0.049, -0.198, -0.209, -0.433, 0.053, 0.408, -0.065, -0.298, 0.132, -0.55, -0.113, -0.12, 0.347, 0.042, -0.045, -0.123, 0.265]

Llama:
[0.11, -0.017, 0.538, 0.567, -0.468, -0.324, -0.07, 0.575, -0.075, 0.115, 0.651, -0.428, -0.495, -0.097, 0.554, -0.036, -0.01, 0.026, -0.304, 0.867, 0.345, 0.722, -0.043, 0.105, 0.205, -0.24, -0.213, 0.094, 0.384, 0.003, 0.317, -1.571, -0.392, -0.524, -0.106, 0.013, 0.558, 0.108, 0.249, -0.037, -0.109, -0.313, 0.709, 0.34, -0.17, 0.094, -0.227, -0.359, 0.237, 0.242, -0.357, 0.022, -0.02, 0.321, -0.109, -0.033, -0.197, 0.288, -0.184, 0.007, 0.554, -0.384, 0.112, 0.217, -0.547, 0.318, -0.167, 0.643, 0.073, 0.078, -0.116, 0.009, 0.158, 0.022, 0.09, 0.101, 0.017, 0.111, 0.425, 0.503, -0.154, 0.159, -0.516, 0.269, -0.966, -0.259, -0.552, 0.049, -0.344, -0.627, -0.019, 0.554, -0.471, 0.372, 0.48, -0.079, 0.15, 0.387, -0.115, 0.273, 0.248, -0.591, -0.157, 0.196, 0.588, 0.074, 0.03, -0.119, -0.353, 0.033, -0.451, -0.358, -0.174, -0.02, -0.258, -0.237, -0.263, -0.382, 0.063, -0.548, -0.269, 0.06, 0.338, -0.182, -0.211, -0.215, -0.024, -0.138, -0.082, -0.571, 0.571, -0.535, 0.771, -0.813, -0.625, 0.124, -0.067, 0.169, -0.561, 0.608, 0.229, -0.154, 0.146, 0.289, 0.072, -0.467, 0.033, 0.183, -0.347, -0.178, -0.357, 0.091, -0.411, 0.355, -0.077, -0.509, 0.485, 0.213, -0.428, 0.497, -0.71, 0.37, 0.009, -0.24, -0.515, 0.627, -0.269, -0.36, -0.609, 0.029, -0.327, 0.067, 0.286, -0.19, 0.271, 0.033, 0.131, -0.471, -0.166, -0.2, -0.125, 0.694, -0.019, -0.294, -0.135, 0.174, 0.078, 0.014, -0.529, 0.191, -0.372, 0.417, 0.258, 0.27, 0.092, -0.826, 0.625, 0.167, 0.351, -0.011, -0.101, 0.085, -0.074, -0.489, -0.248, -0.434, -0.097, 0.081, 0.052, -0.127, 0.399, -0.531, -0.222, 0.658, -0.377, -0.222, -0.11, 0.756, 0.253, 0.306, -0.113, 0.153, 0.806, 0.67, -0.253, 0.53, 0.784, 0.027, 0.073, 0.043, -0.05, 0.234, -0.439, -0.488, -0.118, 0.244, -0.3, -0.196, 0.362, -0.175, -0.317, -0.251, -0.168, 0.327, -0.755, 0.227, -0.539, -0.822, -0.15, 0.469, 0.098, -0.127, 0.732, -0.159, -0.725, 0.006, 0.15, 0.018, 0.177, -0.355, -0.327, -0.123, -0.598, -0.526, -0.575, -0.084, -0.267, -0.249, -0.217, -0.051, 0.261, -0.378, 0.383, -0.178, -0.455, 0.208, 0.178, 0.009, 0.048, 0.282, -0.597, -0.345, 0.006, -0.069, 0.033, 0.309, 0.373, 0.18, 0.427, -0.373, 0.34, -0.585, 0.106, -0.159, 0.048, -0.163, -0.18, 0.203, -0.068, 0.321, 0.46, 0.135, -0.154, 0.084, -0.408, 0.254, 0.019, -0.147, 0.325, 0.187, -0.185, 0.106, 0.207, -0.14, 0.183, -0.702, 0.158, 0.059, 0.182, -0.741, 0.24, -0.936, 0.332, 0.322, -0.345, -0.281, 0.197, -0.035, -0.008, -0.414, -0.7, -0.266, -0.134, 0.063, 0.07, -0.245, -0.221, 0.251, -0.092, -0.294, -0.83, -0.567, -0.006, 0.488, -0.145, 0.203, 0.091, 0.534, 0.024, 0.367, -0.176, 0.237, 0.211, 0.16, 0.186, 0.191, 0.462, -0.437, -0.34, -0.147, -0.099, -0.124, 0.105, 0.526, -0.12, 0.433, -0.418, -0.229, 0.214, 0.337, -0.377, 0.322, -0.278, 0.283, -0.64, 0.596, 0.411, 0.356, 0.481, 0.204, 0.119, 0.577, 0.026, -0.818, 0.396, 0.248, 0.315, -0.391, -0.659, -0.558, 0.607, -0.46, 0.491, 0.344, -0.364, -0.013, 0.462, 0.988, -0.412, -0.354, 0.254, -0.201, -0.497, -0.255, 0.22, 0.377, -0.678, -0.082, 0.13, -0.167, 0.384, -0.51, 1.023, 0.079, 0.365, -0.363, 0.122, 0.062, 0.037, 0.172, -0.036, -0.12, -0.035, -0.209, 0.195, 0.462, 0.262, -0.566, -0.014, -1.055, 0.109, 0.387, -0.121, 0.308, -0.0, -0.113, -0.288, 0.313, 0.087, 0.1, 0.219, 0.482, -0.104, -0.095, -0.708, 0.128, 0.229, 0.328, 0.041, -0.313, -0.209, -0.488, -0.452, -0.181, 0.027, 0.184, 0.297, 0.201, 0.309, 0.151, 0.167, -0.169, -0.015, 0.047, 0.172, 0.125, 0.374, 0.681, -0.075, -0.695, 0.126, -0.035, 0.199, 0.346, -0.075, 0.25, 0.837, 0.059, 0.193, -0.564, -0.027, 0.097, -0.061, -0.237, -0.056, 0.554, 0.073, -0.155, -0.822, -0.251, -0.218, 0.506, 0.781, -0.572, 0.101, -0.284, -0.546, -0.475, -0.061, -0.339, 0.405, 0.282, -0.338, 0.639, 0.114, -0.078, 0.069, -0.233, 0.066, 0.407, -0.395, 0.178, 0.113, -0.307, -0.501, 0.348, 0.118, 0.584, -0.191, -0.485, -0.203, 0.143, -0.056, -0.003, 0.173, -0.507, -0.482, -0.225, -0.442, -0.37, 0.641, -0.606, -0.123, 0.13, -0.268, -0.33, 0.025, 0.39, -0.238, -0.193, 0.107, 0.126, 0.316, -0.386, -0.446, -0.036, -0.304, 0.031, 0.33, 0.244, 0.163, 0.03, -0.042, -0.261, 0.381, -0.305, -0.053, -0.291, 0.249, -0.297, 0.013, -0.03, 0.44, -0.133, -0.198, 0.495, -0.228, -0.224, 0.079, -0.102, -0.344, -0.084, -0.236, -0.998, 0.447, 0.375, 0.568, -0.241, 0.267, -0.053, 0.036, -0.265, -0.152, -0.244, -0.556, 0.372, 0.423, -0.298, 0.196, 0.028, 0.251, 0.377, 0.333, 0.111, -0.171, 0.164, -0.302, -0.047, -0.409, 0.222, -0.07, -0.038, 0.261, -0.331, 0.225, 0.443, -0.354, -0.248, 0.03, -0.013, 0.035, 0.452, -0.296, -0.823, 0.244, -0.029, -0.311, 0.225, -0.002, 0.006, 0.065, -0.146, -0.207, 0.271, 0.187, 0.329, -0.181, -0.378, -0.052, 0.4, 0.475, -0.114, 0.489, 0.064, -0.146, -0.124, 0.168, -0.039, -0.095, -0.347, 0.218, -0.082, 0.118, -0.321, 0.541, 0.116, -0.381, 0.19, 0.438, -0.247, -0.263, 0.012, 0.012, 0.056, 0.28, 0.221, -0.316, -0.136, -0.413, 0.096, -0.036, -0.189, -0.037, -0.115, 0.301, 0.106, -0.045, 0.201, 0.165, 0.058, -0.22, 0.739, 0.168, 0.064, 0.198, 0.242, 0.248, -0.478, -0.247, 0.301, 0.134, 0.451, -0.22, -0.062, -0.246, 0.025, 0.323, 0.145, -0.057, -0.107, -0.53, 0.195, 0.426, 0.178, 0.502, -0.153, 0.243, 0.06, 0.095, -0.577, 0.202, 0.328, -0.04, -0.292, -0.204, -0.189, -0.102, 0.309, 0.378, 0.056, 0.304, 0.05, -0.213, 0.103, -0.105, 0.051, 0.093, 0.176, 0.267, -0.322, 0.038, 0.301, -0.259, -0.441, 0.402, -0.019, -0.198, -0.062, -0.28, 0.265, -0.195, -0.142, 0.275, 0.571, 0.163, -0.022, -0.564, -0.157, -0.518, -0.548, -0.276, -0.017, 0.099, 0.215, 0.302, 0.227, 0.389, 0.323, -0.195, -0.339, -0.373, 0.338, 0.15, 0.14, -0.144, -0.796, 0.003, -0.312, 0.27, 0.591, 0.044, 0.6, -0.398, -0.416, -0.161, 0.257, -0.059, -0.26, -0.041, -0.183, -0.023, -0.684, -0.137, -0.242, 0.455, -0.076, -0.406, -0.127, 0.064, 0.145, 0.047, -0.594, -0.007, 0.35, -0.513, -0.293, 0.188, 0.056, -0.064, -0.596, -0.452, 0.088, -0.192, 0.208, 0.254, -0.087, 0.023, -1.771, -0.288, 0.279, 0.109, -0.242, 0.12, 0.224, 0.594, 0.455, -0.243, 0.192, -0.368, -0.277, 0.343, 0.333, -0.407, 0.005, 0.248, -0.173, 0.291, -0.467, 0.103, -0.168, 0.038, -0.102, 0.175, -0.305, 0.389, -0.353, 0.081, 0.279, 0.137, -0.357, 0.321, 0.116, -0.147, 0.292, 0.078, -0.005, 0.064, -0.248, -0.337, -0.129, 0.078, 0.04, -0.069, 0.037, 0.087, 0.021, 0.185, 0.24, 0.065, 0.049, -0.341, 0.095, 0.196, -0.189, -0.096, 0.221, 0.141, 0.273, -0.326, -0.493, -0.33, -0.143, -0.176, -0.3, 0.246, -0.479, -0.151, -0.007, 0.072, -0.129, 0.223, 0.107, -0.539, -0.178, 0.319, 0.047, -0.218, 0.214, -0.048, -0.266, -0.096, 0.187, -0.018, -0.041, 0.049, -0.158, -0.004, 0.77, 0.263, 0.425, 0.412, -0.107, 0.106, -0.436, -0.052, 0.211, -0.042, 0.368, 0.084, -0.144, -0.325, -0.111, -0.132, 0.432, 0.059, 0.104, -0.121, 0.469, -0.185, -0.122, 0.028, -0.552, 0.231, 0.324, 0.567, 0.227, 0.279, 0.535, -0.256, -0.323, -0.157, 0.116, -0.108, -0.176, 0.171, 0.325, -0.189, 0.252, 0.608, -0.071, 0.334, 0.26, -0.176, 0.366, 0.529, -0.178, 0.053, -0.038, -0.145, -0.021, 0.298, -0.368, 0.13, -0.243, 0.094, 0.024, -0.224, 0.08, 0.014, -0.083, 0.493, -0.354, 0.539, 0.012, 0.395, -0.187, 0.418, 0.059, 0.444, 0.081, -0.206, 0.348, 0.215, 0.107, -0.013, 0.486, -0.094, 0.143, 0.29, 0.042, -0.156, 0.059, 0.105, 0.452, -0.476, 0.008, 0.336, 0.304, 0.258, 0.027, 0.236, 0.662, 0.121, 0.333, 0.238, 0.58, 0.345, 0.126, 0.067, 0.2, 0.063, 0.011, -0.241, 0.197, 0.152, 0.025, -0.39, -0.917, 0.051, 0.337, -0.09, -0.126, 0.239, 0.147, -0.195, -0.141, 0.115, -0.318, -0.29, -0.352, 0.277, 0.172, 0.086, 0.113, -0.227, -0.527, 0.416, -0.263, -0.572, 0.005, 0.341, -0.318, 0.232, -0.134, -0.365, 0.369, 0.465, -0.063, 0.892, 0.559, 0.395, 0.006, -0.476, 0.127, 0.146, 0.763, -0.075, -0.417, -0.561, 0.215, -0.586, -0.239, -0.118, 0.282, -0.424, 0.476, -0.416, -0.23, 0.278, 0.217, -0.284, 0.126, 0.114, -0.199, 0.201, 0.244, 0.138, -0.266, -0.064, -0.044, -0.055, -0.464, -0.022, -0.502, 0.189, -0.011, -0.22, -0.361, -0.098, 0.477, 0.309, 0.039, 0.239, 0.546, -0.358, 0.003, -0.335, -0.194, 0.535, 0.479, 0.38, 0.173, 0.104, -0.153, -0.327, 0.088, 0.147, -0.217, -0.146, -0.316, -0.278, -0.119, -0.335, -0.175, 0.037, 0.252, 0.353, 0.39, -0.119, 0.044, -0.056, 0.237, -0.18, -0.251, 0.054, -0.392, -0.298, 0.16, 0.174, -0.047, 0.21, 0.283, 0.084, 0.057, -0.073, 0.288, -0.17, -0.253, 0.013, -0.058, -0.485, -0.279, -0.183, 0.193, -0.002, -0.075, 0.359, -0.234, 0.078, -0.075, -0.257, -0.325, 0.214, -0.023, 0.147, -0.059, 0.464, -0.402, -0.411, 0.275, 0.563, -0.038, -0.384, 0.394, 0.233, -0.074, 0.189, 2.911, -0.336, -0.187, 0.091, 0.205, 0.268, -0.259, 0.015, -0.123, 0.232, -0.248, -0.067, 0.291, -0.078, -0.182, -0.012, -0.265, 0.054, -0.213, -0.212, -0.362, -0.333, -0.517, 0.256, -0.164, -0.178, 0.315, -0.008, 0.249, -0.105, 0.074, 0.174, 0.153, 0.092, -0.354, 0.337, -0.292, -0.078, 0.149, 0.288, -0.369, 0.18, 0.325, -0.307, 0.176, 0.011, -0.307, -0.282, -0.139, -0.13, -0.223, -0.081, 0.098, 0.201, 0.09, 0.155, -0.088, -0.123, 0.887, 0.338, 0.065, -0.15, -0.322, -0.08, 0.088, 0.359, -0.473, -0.324, 0.077, 0.016, -0.013, 0.072, -0.579, -0.055, 0.267, 0.355, 0.184, -0.575, 0.071, -0.577, -0.458, -0.217, 0.474, 0.065, 0.202, -0.461, -0.292, 0.386, 0.217, 0.253, -0.111, 0.28, 0.368, -0.261, 0.152, 0.204, -0.018, 0.256, -0.163, 0.239, 0.109, -0.027, -0.487, 0.331, -0.145, -0.405, 0.143, -0.441, -0.217, 0.008, 0.426, -0.103, -0.099, -0.312, 0.294, -0.136, -0.039, 0.041, 0.057, -0.267, -0.138, 0.261, 0.495, -0.422, 0.152, 0.18, 0.31, 0.36, -0.035, -0.231, 0.306, 0.369, 0.524, 0.26, 0.618, 0.189, 0.093, -0.093, 0.241, 0.032, -0.37, 0.167, 0.075, -0.534, 0.203, 0.234, -0.212, -0.054, -0.047, 0.209, 0.029, 0.399, -0.348, 0.288, 0.27, 0.003, -0.205, -0.181, -0.523, -0.091, -0.258, 0.221, 0.057, -0.059, 0.317, 0.255, -0.073, -0.017, -0.56, -0.205, 0.365, 0.124, -0.071, 0.021, 0.377, -0.412, -0.076, 0.298, -0.333, 0.051, -0.108, -0.352, -0.059, 0.19, -0.188, 0.111, -0.303, -0.55, 0.058, 0.109, -0.048, -0.151, 0.231, -0.05, -0.615, 0.439, 0.151, -0.035, -0.035, -0.011, 0.011, -0.183, -0.115, 0.091, -0.248, -0.227, 0.523, 0.22, 0.129, 0.118, 0.278, -0.183, 0.078, 0.521, 0.277, 0.445, 0.101, 0.226, -0.003, -0.119, -0.184, -0.279, 0.002, -0.093, -0.451, -0.093, 0.115, -0.536, -0.099, 0.22, 0.215, -0.024, 0.222, -0.216, 0.143, 0.313, -0.688, -0.162, -0.076, 0.232, -0.2, 0.007, -0.372, 0.124, -0.049, 0.414, 0.624, -0.858, 0.112, 0.163, 0.307, 0.238, 0.076, -0.097, -0.501, 0.319, 0.095, 0.024, 0.093, -0.266, 0.183, 0.403, 0.214, 0.05, -0.205, -0.049, -0.58, -0.364, -0.372, 0.189, -0.484, 0.121, -0.32, -0.071, -0.453, 0.157, 0.065, 0.418, 0.499, -0.15, -0.118, -0.269, -0.268, -0.364, -0.037, -0.246, 0.03, 0.417, -0.494, -0.318, 0.357, -0.092, -0.12, 0.19, 0.362, 0.178, -0.053, -0.191, -0.262, 0.018, 0.178, 0.066, -0.397, -0.111, 0.141, 0.483, -0.029, 0.403, -0.112, 0.006, 0.353, -0.289, -0.152, -0.052, -0.193, 0.446, 0.057, 0.12, -0.013, 0.412, 0.354, 0.143, 0.369, 0.3, 0.445, -0.14, -0.34, 0.069, 0.161, -0.517, 0.326, 0.006, 0.415, -0.086, 0.453, -0.483, -0.329, 0.267, 0.022, -0.703, -0.223, -0.019, -0.005, 0.458, -0.098, 0.025, 0.406, 0.375, 0.031, 0.03, 0.248, 0.002, -0.285, -0.014, -0.161, 0.397, 0.555, -0.446, -0.022, 0.19, -0.113, -0.158, -0.303, 0.501, 0.199, 0.104, -0.377, -0.139, -0.223, 0.332, -0.382, 0.201, -0.008, -0.236, -0.138, -0.304, 0.077, 0.393, -0.101, -0.232, 0.058, -0.65, -0.146, -0.154, 0.476, 0.134, -0.091, -0.162, 0.376]